User Notifications

The requirement for a notification area came up when I was implementing Event-Role invites feature. For not-existing users that were not registered in our system, an email with a modified sign-up link was sent. So just after the user signs up, he will be accepted as that particular role. Now for users that were already registered to our platform a dedicated area was needed to let the user know that he has been invited to be a role at an event. Similar areas were needed for Session invites, Call for papers, etc. To take care of these we thought of implementing a separate notifications area for the user, where such messages could be sent to registered users. Issue

Base Model

I kept base db model for a user notification very basic. It had a user field that would be a Foreign key to a User class object. title and message would contain the actual data that the user would read. message can contain HTML tags, so if someone wants to display the notification with some markup he could store that in the message. The user might also want to know when a notification was received. The received_at field stores a datetime object for the same purpose.

There is also has_read field that was later added. It stores a boolean value that tells if the user has marked the notification as Read.

class Notification(db.Model):
    """
    Model for storing user notifications.
    """

    id = db.Column(db.Integer, primary_key=True)

    user_id = db.Column(db.Integer, db.ForeignKey('user.id'))
    user = db.relationship('User', backref='notifications')

    title = db.Column(db.String)
    message = db.Column(db.Text)
    action = db.Column(db.String)
    received_at = db.Column(db.DateTime)
    has_read = db.Column(db.Boolean)

    def __init__(self,
                 user,
                 title,
                 message,
                 action,
                 received_at,
                 has_read=False):
        self.user = user
        self.title = title
        self.message = message
        self.action = action
        self.received_at = received_at
        self.has_read = has_read

action field helps the Admin identify the notification. Like if it is a message for Session Schedule change or an Event-Role invite. When a notification is logged, the administrator could tell what exactly the message is for.

Unread Notification Count

The user must be informed if he has received a notification. This info must be available at every page so he doesn’t have to switch over to the notification area to check for new ones. A notification icon at the navbar perhaps.

Screenshot from 2016-07-19 02:00:47

The data about this notification count had to be available at the navbar template at every page. I decided to define it as a method in the User class. This way it could be displayed using the User object. So if the user was authenticated, the icon with the notification count could be displayed.

class User(db.Model):
    """User model class
    """
    # other stuff
    
    def get_unread_notif_count(self):
        return len(Notification.query.filter_by(user=self,
                                                has_read=False).all())
{% if current_user.is_authenticated %}
    <!-- other stuff -->

    <li>
             <a class="info-number" href="{{ url_for('profile.notifications_view') }}">
                  <i class="fa fa-envelope-o"></i>
                  <span class="badge bg-green">{{ current_user.get_unread_notif_count() | default('', true) }}</span>
             </a>
    </li>

    <!-- other stuff -->

{% endif %}

If the count is zero, count number is not displayed.

Possible Enhancement

The notification count comes with the HTML generated by the template at the server. So to check for new notifications the user must either refresh the page or travel to another page. To show newly received notifications without refreshing the page the WebSocket API can be used. I’ve it in my bucket list and I’ll implement it soon.

Advertisements
User Notifications

Getting fired up with Firebase Database

As you might’ve noticed, in my Open Event Android Project, we are asking the user to enter his/her details and then using these details at the backend for generating the app according to his/her needs.

One thing to wonder is how did we transmit the details from webpage to the server.

Well, this is where Firebase comes to the rescue one more time!

If you’ve read my previous post on Firebase Storage, you might have started to appreciate what an awesometastic service Firebase is.

So without any further adieu, lets get started with this.

Step 1 :

Add your project to Firebase from the console.

 newProj
Click on the Blue button

Step 2 :

Add Firebase to your webapp

Open the project, you’ve just created and click on the bright red button that says, “ Add Firebase to your web app”

 addFirebase

Copy the contents from here and paste it after your HTML code.

Step 3 :

Next up, navigate to the Database section in your console and move to the Rules tab.

 screenshot-area-2016-07-18-204133.png

For now, let us edit the rules to allow anyone to read and write to the database.

 screenshot-area-2016-07-18-204437

Almost all set up now.

Step 4 :

Modify the HTML to allow entering data by the user

This looks something like this :

Now let us setup our javascript to extract this data and store this in Firebase Database.

We are almost finished with uploading the data to the database.

Enter data inside the fields and press submit.

If everything went well, you will be able to see the newly entered data inside your database.

screenshot-area-2016-07-18-205651.png

Now on to retrieving this data on the server.

Our backend runs on a python script, so we have a library known as python-firebase which helps us easily fetch the data stored in the Firebase database.

The code for it goes like this

The data will be returned in JSON format, so you can manipulate and store it as you wish.

Well, that’s it!

You now know how to store and retrieve data to and from Firebase.
It makes the work a lot simpler as there is no Database schema or tables that need to be defined, firebase handles this on its own.

I hope that you found this tutorial helpful, and if you have any doubts regarding this feel free to comment down below, I would love to help you out.

Cheers.

Getting fired up with Firebase Database

Uploading json assets and icons to your app via the app generator

If you have tried out our app generator webpage, you should’ve noticed an option that allows you to upload a zip file which will contain the json for the event.

Why do we need this?

Well, this is needed because not every event organizer can maintain a server and API endpoints which contain the details for their event, so they can simply generate a json for the event by exporting it through the options provided to them on Google Spreadsheets and then they can upload them on the server, so that these files can be packaged in the android and web apps.

Implementation :

The implementation is pretty straightforward and consists of 3 parts :

  1. Making changes to the html file to allow user to upload the zip
<tr>
  <td valign="top">
    <label for="sessions">Zip containing .json files</label>
  </td>
  <td valign="top">
    <input accept=".zip" type="file" id="uploadZip" name="sessions">
  </td>
</tr>

2. Retrieve this file in the javascript and then make an AJAX call to the server

var file_data = $('#uploadZip').prop('files')[0];
 var form_data = new FormData();                 
 form_data.append('file', file_data);
 $.ajax({
            
                  url: '/upload.php', // point to server-side PHP script
                  cache: false,
                  contentType: false,
                  processData: false,
                  data: form_data,                         
                  type: 'post',
                  success: function(php_script_response){
                    // do something
                  }
                });

So here, the form_data contains the details about the file to be uploaded.

In the AJAX call, upload.php takes reads form_data variable and then initiates the upload to the server.

3. Setup a PHP script on the server to respond to the above AJAX call

<?php
    if ( 0 < $_FILES['file']['error'] ) {
        echo 'Error: ' . $_FILES['file']['error'] . '<br>';
    }
    else {
        move_uploaded_file($_FILES['file']['tmp_name'], "/var/www/html/uploads/upload.zip"); 
    }
?>

Here in the PHP script, the file is read and uploaded to a temporary directory in the server.

We then manually copy it to a location and name of our choice.

In case there are multiple users accessing the website and uploading their assets at the same time, we need to pass a timestamp variable to the AJAX call to and later on use it while renaming the uploaded file.

This is to ensure that the file uploaded by one user is not overwritten by another user.

How do we use this data during app compilation

The uploaded zip is then uncompressed and its contents are moved to the assets folder of the android app’s directory.

zip_ref = zipfile.ZipFile(path_to_zip_file, 'r')
zip_ref.extractall(directory)
zip_ref.close()
#TODO: Change path here
for f in os.listdir(directory+ "/zip"):
	if f.endswith('.json'):
		copyfile(f, directoy + "open-event-android/android/app/src/main/assets/"+f)
	elif f.endswith('.png'):
		copyfile(f, directory + "open-event-android/android/app/src/main/res/drawable"+f)
replace(directory+"/open-event-android/android/app/src/main/res/values/strings.xml", 'mipmap/ic_launcher', 'drawable/' + f)

Here replace is a function that searches in the source file for the phrase supplied as it’s argument and changes it with the new phrase.

Now when the app is compiled and ran on a user’s device, it will first search for a json file in the assets directory and if it exists, use that for fetching the data instead of making a network call. For this I have used Gson to first parse the offline files otherwise retrofit makes request to the api and fetches the data from there.

But if there is no json in the assets folder, a normal network call using retrofit will be made and the data will be fetched from the API defined by the user.

Uploading json assets and icons to your app via the app generator

Google Maps Api

OpenEvent uses it everywhere. Let me explain it.

The first problem that we faced was user current location.

We wanted to get location based on geolocation method then display this results on the map.

To get geolocation we use HTML5. Take a look on geolocate() function

 

function geolocate() {

        if (navigator.geolocation) {

            navigator.geolocation.getCurrentPosition(function (position) {

            }

        }

    }

 

From variable position you can get longitude and latitude params. These values allow us to move marker on the map on right position and get the name of user current location. Based on these data we are able to find the closest events and display it in a distance order.

To get information about city or country location you have to execute a simple GET request to google MAPS. In that case you need:

 

curl http://maps.googleapis.com/maps/api/geocode/json?latlng=17.0112,15.06

 

above result returns json file where you can find name of location: Nguigmi, Niger

 

Then, we faced a new challenge- we had to choose which map would be better for OpenEvent – Open Street Map or Google Maps. Finally we have chosen google maps because it wa more comfortable for our team.

 

OpenEvent simple searching engine is based on latitude and longitude params. So we transform all requests which contain city or country names to longitude and latitude params using Google Api. It allows us to avoid having problems with different locations’ names which occur in different nationalities.

 

Google Maps Api

Adding Client Side validation to Login and Registration Forms

Its very important to have a client side validation apart from a server side validation. The server side validation only helps the developers but not the clients. Thus it was necessary to add a client side validation to the Login page and its other components so that the users  would be comfortable in this.

I had never before done this and was looking at how to achieve this using jQuery or JavaScript when I cam to know that we were already using an amazing validation tool : Bootstrap Validator

It just involves wrapping the form in the html with the validator plugin and all the checks are automatically carried out by it.

 

1.png

As you can see in the above image, we have just added a data-toggle=”validator” line to the form which automatically wraps the form with the plugin. The above form is the Create New Password form which checks whether the new password and the password entered again are matching or not. If not then the line ,

data-match="#new_password"

checks it with the password in the new_password field and gives an error on the page to the client defined by the following,

data-error="Passwords do not match

Thus adding such checks to the form becomes very simple rather than using jQuery for it. Similarly it was important to add validation to the Register page.

 <input type="email" name="email" class="form-control" id="email" 
 placeholder="Email" data-remote="{{ url_for('admin.check_duplicate_email') }}"
 data-remote-error="Email Address already exists" required="">

This is the validation for the email field. Apart from checking whether the text entered by the user is an email id or not it was also important to check whether the email entered by the user exists in the database or not.

data-remote="{{ url_for('admin.check_duplicate_email')

This line calls a view function to check whether the email entered by the user is duplicate or unique. Here is the function

2.png

This function takes the email value from the request.args and then performs a simple check in the db to check for duplicate email. If the user doesnt exist then the validator receives a simple string “200 OK”. If the error is 404 then it gives the error defined by,

data-remote-error="Email Address already exists"

However this error only for the data-remote part. If the error is something else then it is handled by ,

class="help-block with-errors"

Thus without the hassle of using Ajax and JQuery we can easily add validation to the forms using Bootstrap Validator.

Adding Client Side validation to Login and Registration Forms

How to parse json assets with gson

So most of us have json assets in our app which we parse on runtime to get the data and use it accordingly but what I have seen is that most of the people create a JSONObject or JSONArray after reading the json into an inputstream but then handling it becomes difficult since we have to manually extract every entity in each array which makes it bound to a lot of errors. A better approach to using it is making use of gson : an open source library by Google to serialise and deserialise Java object to (and from) Json. It’s pretty easy to use and makes the development process easy. For those of you not still convinced on using gson, I’d like to demonstrate the code we had to write to without using gson and the one using gson as well.

So to start with lets see the json file we’ll be using. It’s the events.json file from the open event project.

{
  "events": [
    {
      "color": "#fdfdfd",
      "email": "dev@fossasia.org",
      "end_time": "2015-07-14T00:00:00",
      "id": 4,
      "latitude": 37.783839,
      "location_name": "Moscone centre",
      "logo": "http://mysecureshell.readthedocs.org/en/latest/_images/logo_redhat.png",
      "longitude": -122.400546,
      "name": "FOSSASIA",
      "slogan": "Fossasia",
      "start_time": "2015-05-28T13:00:00",
      "url": "www.google.com"
    }
  ]
}

As you can see it has an object that has an array of event objects. So what we’ll first do is that we’ll get the whole json as a string by openeing an inputstream and then directing it to a buffer. Then we convert the buffer array to a string object.

String json = null;
try {
    InputStream inputStream = getAssets().open("events.json");
    int size = inputStream.available();
    byte[] buffer = new byte[size];
    inputStream.read(buffer);
    inputStream.close();
    json = new String(buffer, "UTF-8");

} catch (IOException e) {
    e.printStackTrace();
}

Now we have the json as a string which we can now parse it using a combination of JSONObject and JSONArray. First we’ll access data in the outer json object i.e. “events”. That’ll be done by

JSONObject jsonObject = new JSONObject(json);
JSONArray events = jsonObject.getJSONArray("events");

Now that we have the array, we can traverse it to get the objects inside events array

for (int j=0; j < events.length(); j++){
    JSONObject cit = events.getJSONObject(j);
    String color = jsonObject.getString("color");
    String email = jsonObject.getString("email"); 
    String endTime = jsonObject.getString("end_time");
    String id = jsonObject.getString("id");
    String latitude = jsonObject.getString("latitude");
    String locationName = jsonObject.getString("location_name");
    String logo = jsonObject.getString("logo");
    String longitude = jsonObject.getString("longitude");
    String name = jsonObject.getString("name");
    String slogan = jsonObject.getString("slogan");
    String startTime = jsonObject.getString("start_time");
    String url = jsonObject.getString("url");
}

This is how we go about it. Now for the exiting part.

We already have an Event data class which has the constructor, getters and setters etc.

public class Event {

    int id;

    String name;

    String email;

    String color;

    String logo;

    @SerializedName("start_time")
    String start;

    @SerializedName("end_time")
    String end;

    float latitude;

    float longitude;

    @SerializedName("location_name")
    String locationName;

    String url;

    String slogan;

    public Event(int id, String name, String email, String color, String logo, String start,
                 String end, float latitude, float longitude, String locationName, String url, String slogan) {
        this.id = id;
        this.name = name;
        this.email = email;
        this.color = color;
        this.logo = logo;
        this.start = start;
        this.end = end;
        this.latitude = latitude;
        this.longitude = longitude;
        this.locationName = locationName;
        this.url = url;
        this.slogan = slogan;
    }

    public String getEmail() {
        return email;
    }

    public void setEmail(String email) {
        this.email = email;
    }

    public String getColor() {
        return color;
    }

    public void setColor(String color) {
        this.color = color;
    }

    public String getUrl() {
        return url;
    }

    public void setUrl(String url) {
        this.url = url;
    }

    public String getSlogan() {
        return slogan;
    }

    public void setSlogan(String slogan) {
        this.slogan = slogan;
    }

    public int getId() {

        return id;

    }

    public void setId(int id) {
        this.id = id;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getLogo() {
        return logo;
    }

    public void setLogo(String logo) {
        this.logo = logo;
    }

    public String getStart() {
        return start;
    }

    public void setStart(String start) {
        this.start = start;
    }

    public String getEnd() {
        return end;
    }

    public void setEnd(String end) {
        this.end = end;
    }

    public float getLatitude() {
        return latitude;
    }

    public void setLatitude(float latitude) {
        this.latitude = latitude;
    }

    public float getLongitude() {
        return longitude;
    }

    public void setLongitude(float longitude) {
        this.longitude = longitude;
    }

    public String getLocationName() {
        return locationName;
    }

    public void setLocationName(String locationName) {
        this.locationName = locationName;
    }

}

Now here we name the parameters to the same as that in the json we have or we can just add @SerializedName(entity_name). Then we go to the code for actually retrieving the data from the json file using this data class. How we can do that is by first making a class that’ll get the array of events for us.

public class EventResponseList {
    @SerializedName("events")
    public List<Event> event;
}

Now all we do is

EventResponseList eventResponseList = gson.fromJson(json, EventResponseList.class);

We have a list of events that were in the json array.

Voila! That’s it. It’s so easy to get a list of all the events from the JSONArray and since the library is available for gradle, it’s even better. You can just add

compile 'com.google.code.gson:gson:2.7'

to your build.gradle dependencies and you’re good to go. Cheers!

How to parse json assets with gson

Optimizing page load time

The average size of a web page has been growing at an accelerating rate over the last few years. Last week, when the open-event webapp was tested, it was very slow to load because of loading 67.2 MB data ( which contained audio and image files ) on the web page.I have taken following steps which are good to read to make any web application load faster.

1. Stop preloading of audio files

The HTML5 audio player loads all audio files by default on the page. To stop this we have to change the code as

<audio controls preload="none">
 <source src="{{audio}}" type="audio/mpeg">
 </audio>

Setting preload option to none help us to load audio only when it is clicked. Hence it decreases the HTTP calls and decreases the page loading time.

Previously

Network tab with Audio files takes 68.0MB  load and 13.6 minute loading time.

fail

Now

29

Network tab with option <audio controls preloaded=”none”> takes 1.1 MB load and  1 minute loading time. This was a huge improvement but more optimizations can be done.

2. Using Compression Middleware

Express compression middleware works like a charm by compressing the response coming to the web page. It is too simple to add.

$ npm install compression
var compression = require('compression')
var express = require('express')

var app = express()

// compress all requests
app.use(compression())

// add all routes

This has compressed the responses and decreased the page load time.

3.  Reduce the number of HTTP requests

Another great way of speeding up your web pages is to simply reduce the number of files that need to be loaded.

Combine files

Combining multiple stylesheets into one file is a really useful way of eliminating extra HTTP requests. This strategy can also be adopted for your JavaScript files. A single larger CSS or JavaScript file will often load quicker because more time can be spent downloading the data rather than establishing multiple connections to a server.

After such optimizations, the webapp loads now in 8-15 seconds with DOM loaded in 2 seconds.

fulloptimize

The result of these optimizations are awesome. We can test the page speed by using various tools like pagespeed, speedtracer etc.

Optimizing page load time