   Introduction Book convention Server setup Telegram and NodeJS Setup the bot in the Telegram Setup certificates The code List of commands Slack and NodeJS Setup the bot in the Slack The Code Messenger and NodeJS Setup the bot in Facebook The code Language data preparation Introduction Wordnet Conclusion Appendix A Appendix B Introduction As in every good book we need to go back in time and start with some history. If not the first, definitely best known bot aka chatbot is a program called ELIZA, created by Joseph Weizenbaum from MIT Artificial Intelligence Lab in 1960s. The program was quite simple, but the effect was immense. People really believed it was a doctor typing on the other side and talked … well, typed to it as a one. (via: http://www.alicebot.org/articles/wallace/eliza.html 23.8.2016) Smartness has always fascinated people so we will be writing smart bots … well, not so fast. The bots we are talking about are much simpler, but they can have smartness of … well, … uhm gold fish or something… Anyway … let’s do it…. We will be doing bots on three major platforms for the communication: Telegram, Facebook’s Messenger and Slack. The technologies are similar, but still a bit different. We are using NodeJS, but the technology is similar if you prefer other programming languages. You need to have access to the internet and some possibility to setup a secure web server. Certificates are not a problem as we will see later. Book convention To make the book easier to use I tried to mark some words that I thought are important bold and also added code that was written or data returned from the server in the boxes. The boxes are coloured depending on the code or data in them.    Server setup We will setup our web server on the Apache platform, but you can, obviously, use other web servers (as Nginx or similar). The only thing we will be using is a proxy passing of communication, so you can setup similar thing with other web servers. Just one more thing. Since you need to use an SSL certificate you will need to have a domain name and not the IP address. Let’s setup an Apache proxy server. The syntax is rather simple: We are creating a https server since the message environments need to use secure communication. As you can see I left email in ServerAdmin as default. You can obviously put your own email. Be careful with the selection of port. Here we are using port 8080, but it depends on the port you are using for your NodeJS server (e.g. if the server is on port 8082, you should use 8082 here). The same setup will be used for all the chat platforms. But remember: you need to create a certificate for the domain. And you might also need to change the port for the localhost server since some traffic might still go to the previous one. Telegram and NodeJS Setup the bot in the Telegram First, you need to set up a bot in Telegram app. You need to connect with BotFather. This is a bot that creates other bots .. how obvious. To create a bot just use a command /newbot and BotFather will explain it to you what you need todo . Let’s give the bot a unique name.. The name is not the same as a username and you can use whatever you want. I guess I don’t need to remind you, that it is better to use descriptive names. You can see the name of the bot as a person’s name and you will have to add username after that. As mentioned, the BotFather leads you through the procedure. When choosing the username you need to add “bot” at the end. And if you choose a name that is already taken, the BotFather lets you know about that. After successfully entering a bot name you will get the success message. With that you will get some explanation about newly created bot and also a secret token, that should really be secret. You can add a description, about section and profile pic to the bot and you can do that via commands /setdescription, /setabouttext, /setuserpic. OK. So this is basically it for creating a bot inside the Telegram. Let’s start coding the server that accepts connection via Telegram and replies with a smart answer. As said in the introduction, we will be using NodeJS. For all our examples we are using request module, so if it is not installed yet, use npm install request. Start creating a bot with this command. The BotFather will lead you through the procedure of creating a new bot. After adding a name for the bot you will need to add username for the bot and you need to add bot at the end, so the users will know this is the bot. Just to be clear - to communicate with the bot you are using its name and not username. Type in the username and you get the following response. Done! Congratulations on your new bot. You will find it at telegram.me/<name_bot>. You can now add a description, about section and profile picture for your bot, see /help for a list of commands. By the way, when you've finished creating your cool bot, ping our Bot Support if you want a better username for it. Just make sure the bot is fully operational before you do this. Use this token to access the HTTP API: <secret token code> For a description of the Bot API, see this page: https://core.telegram.org/bots/api The secret token code is the token you have to keep secret, so don’t publish it no matter what. To get manual on different commands for your bot just click the link on description of the Bot API. Use this command to see the token again. You will have to pick for which bot you would like to see the token, if you have more than one bot. If you forgot your secret token or for some reason need a new one, use this command. You will have to select for which bot you would like to generate a new token. Creating new list of commands for the bot can be done with the following command: “What is the list of commands?”, you ask. Glad you did. We will explain it later. It’s obvious - if you need to change the bot’s name use this command. Describe what the bot is doing. The bot’s profile photo.  To change profile photo of the bot. You can upload the photo by clicking the clip button and selecting the image. Add text that will be visible in command line input box when you type in the name of the bot. When you enable geo location, the Telegram app will ask you for the location every time you try to communicate with the bot. To know which of the provided results your users are sending to their chat partners, send @Botfather the /setinlinefeedback command. With this enabled, you will receive updates on the results chosen by your users.  Please note that this can create load issues for popular bots – you may receive more results than actual requests due to caching (see the cache_time parameter in answerInlineQuery). For these cases, we recommend adjusting the probability setting to receive 1/10, 1/100 or 1/1000 of the results. You need to send the list of commands that will be shown to the users. Go to new line to enter commands by pressing shift + enter.  Simple command to allow or forbid bot to enter the group.  Enable - your bot will only receive messages that either start with the '/' symbol or mention the bot by username. Disable - your bot will receive all messages that people send to groups. Current status is: ENABLED remove the bot, but you need to confirm it with exact phrase “Yes, I am totally sure.” (with a punctuation mark at the end). cancel current operation.  Setup certificates All the bots are working over the secure SSL connection, so you need to provide the certificate. And currently - only Telegram has a possibility for you to use self signed certificate, but you need to upload the public part of the certificate, to let the Telegram trust the certificate. You can do a simple test if you got the proper token even without letting the server know about the certificates and that is getting the bot’s name via curl command: NOTE: Pay attention to bot prefix for the token. You have to use it. We can, of course, create a pure javascript solution: As you can see, we are sending a parameter url by using curly brackets and you can send additional parameters if needed this way. The result (in JSON format) is found in body, if everything goes well :) For other objects and methods you can check the Telegram API documentation. There are two possible ways to get the incoming updates (data from the server) with two different commands: getUpdates and setWebHook. getUpdates uses long polling which basically means: we ask server and then wait. On the other hand you give Telegram an URL via setWebHook and when event happens the Telegram sends POST request to the server and gets the data. Let’s try with getUpdates For the sake of book space we will use curl, but to use NodeJS example just replace getMe in previous example with getUpdates. and we get result in JSON format: So we received nothing. Why? Well, we didn’t start the bot yes. So let’s start the Telegram app and start it. We need to search for the bot username via search input (don’t use name) and select it. After clicking Start we can write a message to the bot. Let’s repeat the curl command to see what we will get. Wooohooo … our first response from the Telegram. As you can see, there is already a /start command in the list. What happens if we repeat the getUpdates command? Let’s try. Ooops, we get the same result all over again. One of the things getUpdates does is just sending results every time you ask for them, so if you want to use that command you will have to check which responses you have already processed. You can set a limit to the number of messages received (default is 100), via parameters in getUpdates. But there is a better way to fetch messages (updates as Telegram people call them) from the Telegram and that is by using command setWebhook. We are not really fetching the messages but rather let Telegram know that whenever something arrives for us, we are ready to receive it. There is one other thing. You can’t use both getUpdates and setWebhook, so by using setWebhook we exclude the possibility to use getUpdates. Let’s first try the Telegram bot by using self-signed certificate. How to create the certificate can be found in Appendix A. Since we are using self-signed certificate we need to let the Telegram know the public key of our certificate. This can be done by sending the public key with the url of our secure server that will be used to create a reply to the user query. The first test would be to use curl: NOTE: We are using @ symbol for file. To set the same thing with NodeJS we will have to use request package:  If everything goes well we will get the response: If we want to turn off the webhook service we just need to send empty request to the api.telegram.org We are now ready to setup the server and try it. But since we will be using “proper” SSL certificate (one of the procedures is also attached in appendix B) we don’t need to upload public key of the certificate so you can set the connection by using only URL. So let’s change WebHook to work with that certificate. Don’t forget to remove the connection with your own signed certificate and WebHook first, by using: And set new connection with the URL: The code We can now setup NodeJS bot server on our server to simply reply to whatever has been sent to it.Since the server is running on the port 8080 we will leave the Apache setup as it is.  Let’s explain the code a bit. The server receives the data from the Telegram. Remember we setup the url for the WebHook. After receiving the data (POST HTTP method) in JSON format it gets parsed to get the id of the chat, so we can reply to the proper message in Telegram and we also parse the text that was sent to the server and setup variable return_text. The value of the variable is returned to Telegram via an API command sendmessage where we need to add chat_id and text values. List of commands No, I didn’t forget. What is list of commands? When communicating with BotFather you can use command /setcommands and select for which bot you would like to add commands. Select one that you have created and the response will say Let’s try with: You need to enter the list in one message and to move to new line you have to use Shift + Enter. By using only Enter you send the command and we don’t want that, right now. When using the command you always need to add slash and name of the command. The whole text will be sent from the Telegram to our bot server in the message body and since the text starts with slash we can assume we are dealing with command. Slack and NodeJS Setup the bot in the Slack You can’t use self-signed certificate with Slack, so we will not go into this and we will just use “properly” signed certificate as shown in Appendix B. Let’s first create a bot in the slack. Bots here are connected to the real time messaging API. So go to Apps & Integrations in your slack client (tested on mac) It takes you to the web page where you can select Build to start building the bot. You will have to decide if the bot will be used only for your team (that you created in Slack) or it will be available to everybody.  If you just want to test it you can create your own team and create a bot there.  Selecting Incoming WebHooks gives you a possibility to post messages from external source to Slack. It’s a HTTP request in JSON format. We’ll select Bots for our usage and look at it a bit later. Slash Commands gives you a possibility to create slash commands and when using customised command the slash will send relevant data to an external URL (e.g. server). Outgoing Webhooks is more of a flag thing - if something is going on, send data to external URL. So let’s look at Bots now. Click on Bots and you can now create a bot username with some restrictions: Usernames must all be lowercase. They cannot be longer than 21 characters and can only contain letters, numbers, periods, hyphens, and underscores. Most people choose to use their first name, last name, nickname, or some combination of those with initials. After selecting the name click Add bot integration and the wizard takes you to the next screen where you can change the name of the bot and other integration settings stuff. But pay attention to the API token. Another thing to note is on which channel is the bot used. After everything is well and done and if you made any change just click Save Integration and you are good to go. Let’s close the web page and setup our slack related bot server. To add a bot to the slack you need to connect it with a custom trigger that will call the create bot. Start by clicking on the Manage selection (by the Build menu), then click Custom Integrations and select Slash Commands. Click on it and start new configuration by clicking on the Add Configuration. Add a command that will act as a trigger to call the bot and don’t forget the slash. In the URL field you have to add the URL of your bot server (it should be in the secure URL format https://..). You can leave other stuff empty or default. The created command will be called /return and the service will do the same as in Telegram. So, it will return what you wrote. Pay attention to the token - if you need to see if the user is really part of your slack team, but we will leave it for now. Finish with clicking Save Integration. After that try to use the command and the bot should respond. You can also check what you get in the command line of the bot. The Code We can now customise settings for the bot with autocomplete text (what will show in command line when you type in the custom command), hints … etc… As you can see the server is similar as in Telegram. We will use Apache again, but this time the NodeJS server will be running on the port 8082, so you need to change the setup procedure for the Apache accordingly. Messenger and NodeJS Setup the bot in Facebook To setup a messenger bot you need to have Facebook account, of course :) Open the Facebook in the browser and select Manage Apps under Explore (on the left side). Here we can “Add a New App”. Let’s click it. Don’t forget to select a category, but you will be notified anyway. After clicking Create App ID we get the Product Setup menu. There are a lot of options you can work on, but we will concentrate on those related to bots and Messenger so let’s choose it. After clicking Get Started we will Setup WebHooks.  Add a Product and select Messenger (Get Started). Click Get Started. Generate token in Token Generation by selecting the page and giving a permission to the app. You will get page access token which you need to copy in the server file. Just click on it and it will be copied to Clipboard. To set up the WebHook you just need to add a url (with https://) and verify token (whatever you pick). Just select all fields under Subscription. Click Verify and Save (the server should be running). You will also need to select a page to subscribe your WebHook to the page events - just pick the same one you used for token generation and Subscribe. The code The code for the server is: In the Messenger we also need to add a GET responder since Facebook checks if server is still running. Other than that, the procedure for the bot server is very similar to previous versions for the Telegram and Slack. We parse received data to get the message sent by the user in variable received_text and use that value as a responder to the user input. Language data preparation Introduction Before you can use the received text data you need to do some housework. To make things easier we will remove some characters that are not that useful for understanding the received text. We can easily remove punctuation and lose very little meaning of the sentence. Next idea is to remove one or all of the following: • 100 most common words in language you are using (be careful since we are working with very short text e.g. sentence and you might remove important information) • remove most re-occurring words in the document (this one is useful for larger texts) • shorter words (e.g. 3 letter words, because they don’t carry that much information)  (from https://en.wikipedia.org/wiki/ Most_common_words_in_English) The next tool we can use is lemmatisation, which means changing the word to its “basic” form and by using a technique using the technique called part-of-speech tagger you can tag the words with their forms and by using verbs you can find out what the user wants… well, usually. One of the useful tools is also a stemmer. The stemmer tries to cut end of the words and group similar words. (e.g. receive, received, receiver … etc. are all cut to the stem receiv. But for all that we could be using some tools, if they are available. A search on the internet shows there is a natural language processing package for NodeJS called Natural. Let’s install it with: As stated in introduction we will pull out words from the received text. We will be using a tool called Tokeniser for that. The code will produce a list of words from a sentence. We can search through the list for 100 most common words in language and remove them. This would mean we are removing a, from which would already cause some troubles, so we will not be using such preprocessing of the text now. As you can see this part of the process involves some thinking and experimenting. Since we would have to setup some external files for a POS tagger used in natural we will not use that tool and instead try to cop with a stemmer. Here is the example for different versions of word receive. Wordnet Since this is not a book on natural language processing we will take a really short look into Wordnet. Official description of it is “A lexical database for English” which basically means you can get synonyms for words. So if your user uses synonyms in their sentences to the bot you can try to add Wordnet to your code and try to find a synonym for the words used. The Wordnet can be found online (https://wordnet.princeton.edu) and is also available for a download. Let’s try our receive example. By searching online we get the following: As we can see, this is a verb of examples of different uses and synonyms that might mean the same. Conclusion This concludes our look into bots by using NodeJS. Hope you found it useful and will be on your way coding bots in NodeJS or any other programming language. The idea is basically the same, so you will not have a lot of problems porting our examples to other programming languages. Appendix A Creating our own private certificate is quite simple. The following command will create the whole thing. Mind you that certificates need to have an expiry date and with this command we are saying that the certificate expires after a year (365 days). To create a longer lasting certificate you can just add some days to the days part. Sometimes I create a certificate valid for 10 years. This might cause some security issues, but it’s ok for testing. Although - if you are testing, why set up a certificate valid for 10 years? The important thing here is that you add proper domain name to the CN part (without https://). The certification procedure will fail if you make a mistake here. After the procedure you will get two files: private and public key in a text format. The private key will start with a BEGIN RSA PRIVATE KEY and public with BEGIN CERTIFICATE. To see what is included in a public certificate you can use following command: Appendix B We can create an open source free ssl certificate by using https://certbot.eff.org  Select which web server you use and the platform and you get info how to setup the certificate. We will select Apache as a web server and since it runs on Debian Linux that will be our Operating System selection. We have to install some packages on our web server to use certbot tool. The installation procedure depends on the operating system you use for your web server, but the command is always the same: If you have many virtual servers setup on the same IP you will get the list of them and you pick one. The name should be the same as the name of the domain server. On Linux the created keys will be located in the directory /etc/letsencrypt/live under the directory with the domain name. To use the certificate on the web server (remember we are using Apache), you have to add fullchain and privkey files.
