Creating the Timeline

04 June, 2020
42 views
The timeline acts like a time machine for people to look back in time and view past statistics
The timeline acts like a time machine for people to look back in time and view past statistics

Preface

We have a lot of people meticulously draft feature requests and raise them on the repository's issues page. After a feature request gets reviewed by maintainers and labeled available, contributors get creative and open Pull Requests against these requests with a working implementation.

Most of the time, it can get a little tempting for me to get started on some of the issues right away if I'm sure I know how to implement it or sometimes to try is just plain fun.

However, that's not always the case because there can be times when I simply don't know how to do something. That's when contributors help by raising Pull Requests, and that's the beauty of open source. I get to work alongside people who have experience in certain things that I don't necessarily have and they help fill this gap.

The feature request

A user raised the timeline feature quite some time back. It wasn't actively worked on because the implementation potentially introduced unnecessary complexity to the codebase since data handling existing at that time was convoluted. Things could've been a lot easier if data handling was done a little differently, which I'll be addressing below.

Issue #135
Issue #135

Everything but the timeline mentioned in the description was already present on covid19india.org.

Only the timeline that one can use to scrub through days and visualize the increase/decrease in the intensity of cases over time needed work.

Investigating the current structure

So now, I know that we need the timeline to function on the home page.

Before I start working on it, I revisit the code and open up the source code for the home component. I look around and remind myself how the component is currently structured and how the data flow takes place.

For the sake of simplicity, I'll be showing them as pictures instead of code.

Components that will be directly affected by the addition of a timeline
Components that will be directly affected by the addition of a timeline

I realize that three significant components show the statistics on the homepage. All of them currently show the current day's statistics, so using a timeline, people can choose a date and view the statistics for that specific day.

notion image

Now I ask myself whether the data to go back in time and refer to past statistics was readily available from the APIs we use, and the answer was a little complicated.

The Timeseries component contains the graphs that already have past statistics available because, of course, they require past statistics to plot the graphs. Then, one can think that this time-series data could then be used in the Table and Map Explorer components as well, right?

Well.

The addition of new data points was carried out iteratively during the initial weeks of development on both the website and the API. First, we started by putting out only state-wise data, then moved to statewise timeseries, and then the rest. As a result, the endpoints were added to the website one by one as they came in from the API, and we had to route it to the appropriate components by mixing and matching data points here and there.

Shoutout to @JunaidBabu for rapidly pushing these endpoints out

For instance, the home component used four endpoints to fetch all the required data needed to be displayed on the home page.

notion image

All of these then get parsed on arrival and supplied as props to each of the components.

notion image

On the other hand, timeseries data had state-level information only, so we couldn't use that in the table that had district-level statistics as well unless we programmed an explicit fallback mechanism. The props to the table and rows were structured differently and wouldn't work with the timeseries data without little modifications to the frontend.

notion image
 

So now we had to support district-level data in timeseries format. That meant we would be adding ~735 districts x 4 statistics x 90 days so far — a whopping 4.2MB worth of data. That is a lot of data to download on initial load, and unfair to those who don't check out past statistics.

notion image

Thirdly, even if were to allow for a large download, with the current structure choosing an arbitrary date on the homepage would re-run the parse, filter out the dates, re-calculate the statistics and supply it as updated props to the components. On selecting a date, there's a significant load in parsing this data repeatedly, which isn't helpful.

Hence, there was a need to re-evaluate our entire component and API structure to get the timeline to work.

Restructuring the API

One of the things we wanted to eliminate early on was to get rid of the parser in the frontend. After we receive a status 200 response, we should ideally be in a state to use the data immediately in our components. Parsing the data to fit the structure of our component on the frontend can be unproductive when we can have that data parsed and already structured in the first place when received from the backend.

We spent a few days studying each component and abstracting a standard data structure so that multiple components can immediately use it without going through another layer of parsing.

Eliminate parsing data in the frontend; have it arrive directly in a usable state
Eliminate parsing data in the frontend; have it arrive directly in a usable state

Next, we wanted to remove as much redundancy as possible in the data so we could have our file sizes small and quickly downloadable. We leveraged the use of optional chaining in javascript to eliminate keys in objects itself instead of having a null value attached to them. An absence of a key simply translates them to zero or null values in the frontend.

notion image

The next way through which we tackled redundancy was to remove calculable secondary data points from the payload. For example, active values can be calculated by using confirmed, recovered, and deceased statistics. Furthermore, in timeseries data, delta values can be calculated by subtracting adjacent cumulative values in O(1); however, it isn't the same the other way round since you'd have to loop an addition operation from 0 → n to reach your cumulative value every time.

While one can argue that we're introducing "parsing" again into the frontend, I guess that's the fine line we draw between using a little of the browser's computing resources and minimizing the download size of the JSON payloads.

But redundancy is not always a a bad thing. Sometimes we can take advantage of redundancy in data by using them as an excuse to build reusable components in the frontend. In the table component, there's no need for a state's row component to be any different than that of a district.

Notice how by using similar data structures for both states and districts, we don't have to handle district rows any differently. We can use a single row component to handle both cases easily
Notice how by using similar data structures for both states and districts, we don't have to handle district rows any differently. We can use a single row component to handle both cases easily

So component reusability (in our case, we've abstracted the cell) would look something like this below.

notion image

Coming back to implementing the timeline, we wanted to make sure that the data sent to the website carried a smaller payload in size with relevant data on initial load. Doing so meant that we had to move from relying on four endpoints to just one single endpoint.

At first, the idea seemed satisfying to arrive at just using one endpoint. However, we realized that we didn't have to fetch a considerable part of the response repeatedly when switching between the dates of the timeline.

The timeseries graphs and maps used this portion of data. One endpoint would've been the icing on the cake (with that said, it's also important to acknowledge that a single source of truth can also be a problem sometimes). But we figured that the timeseries data could reside on a separate endpoint that can be called once and don't have to react to changes in the timeline in terms of fetching new data henceforth.

So to address that, we had to split the final endpoint into two - timeseries.json and data.json. The former gets fetched only once, while the latter gets fetched every time we look back to a different date while using the timeline.

Our restructured endpoints
Our restructured endpoints

Finally, we partitioned data.json by dates in the format of 'data-yyyy-mm-dd.json' to revisit the statistics of any past dates. So now, going back in time was just a matter of changing the endpoint's date.

notion image

Shoutout to @shuklaayush for working on the ultimate parser. Here's my v2 version of the parser at this link, but @shuklaayush beat me to it with his v3 parser by supporting districts and minimizing redundancy :P

Using the newly structured API

When the home component calls this endpoint, it receives the data and distributes it as props to the Table, Timeseries, and MapExplorer components. And if we make these components react to prop changes, then we're golden.

All we have to do is have the user specify a date, then update the endpoint URL and fetch.

When the new data is received, the home component performs a re-render and distributes a new set of props to the components. And since we've made the components to react to these prop changes, the components will have their states updated, and you now have the timeline implemented.

notion image

User interface

So the strategy in terms of code and the API has been figured out. Next comes the user interface.

At this point, I open Figma and have the mobile version of the website open in front of me. I immediately think of all the potential ways I could place the timeline.

notion image
notion image

I try to experiment with every possible way to pick the most straightforward design that would require the least explanation for someone to be able to use it.

 

After a couple of potential options, I ended up deciding to place it here.

Also has the added benefit of users noticing the new feature immediately. (Left: Actions panel, Right: Timeline panel)
Also has the added benefit of users noticing the new feature immediately. (Left: Actions panel, Right: Timeline panel)

That position also had the added benefit of users noticing the new feature immediately. (Left: Actions panel, Right: Timeline panel)

But all of this has just been me moving around rectangles and text boxes on Figma; I still have no clue about how I would end up turning this into code just yet.

(Maybe I had a little idea)

Bringing it alive

I figured out that I would have to switch to the timeline mode by clicking the history icon. So I created a state called isTimelineMode to toggle between the Actions panel and the Timeline panel.

Next, I wanted the toggle to look a little bit more organic rather than just a component replacement. I figured a card flip animation would eventually do the trick. But I didn't know how to implement it at first, so I went around googling for examples for some inspiration.

I remember stumbling upon react-spring, a javascript animation library that uses the concept of spring mechanics to animate elements some time ago and revisited them to see if they had any good examples of a card flip.

And they sure did! I used one of their examples to understand how the library worked and how the flip animation worked and tweaked it to our requirement. While reading the documentation help in learning about extra configurations or get a deep understanding of what the library can do, learning through examples can help a lot.

 
notion image
 

After this, I needed to figure out a way to scroll through the dates. use-gesture is another incredibly well-made package that allows people to interact with components through gestures like drag, hover, scroll, wheel, etcetera. This library, coupled with use-spring, is a powerful combination for creating highly interactive components in React.

Funnily enough, I also used the help of another example on their website to learn how their API works and used it as a starting point to improvise upon it and emulate the functionality I wanted.

notion image

The timeline would behave like a carousel interface, except they are now dates instead of pictures you would see in a typical carousel.

After adding support for dragging on touch devices, I realized that using a pointer to drag while using a touchpad or mouse would be incredibly infuriating for desktop users. So the next step was to support controlling the timeline using the keyboard's LeftArrow to go left, RightArrow to go right, and Escape to reset to the current date. I used the useKeyPress hook available in react-use to add this functionality easily.

Adding a little more detail

By now, most of the work on implementing the timeline is done. At this point, I go over the code and try to abstract everything a bit more to avoid redundancy and profile for any performance bottlenecks to see if any extra re-renders are happening anywhere that I could potentially address and fix before the final merge.

Sometimes in the middle of doing all these things, I try to see if there are any possibilities of squeezing in another little enhancement somewhere.

notion image

Highlights, like the one shown above, was a result of that.

Push to production

After all of this, we create preview deployments and share it internally to test for any bugs that may have slipped during development. After a thorough review, we push to production and hope that nothing crashes🤞🏽