Innocelf Website and Client Admin Dashboard

Table of Contents

Background
#

Innocelf was founded by my spouse in September 2020 as a legal research consultancy, helping innovators navigate a convoluted path of protecting their intellectual property.

Visit Innocelf’s website to learn more and schedule a consultation for your intellectual property needs.

Being a new company, it lacked processes to track key metrics, manage clients, or maintain financial documents for tax purposes. In the beginning, as a one person firm, it was difficult to manage these business related requirements while also maintaining good quality research, i.e. the real service. Clearly, there was a need to create a system to perform these ancillary tasks, while creating a website to help potential clients find us.

Innocelf’s inception coincided with the COVID-19 pandemic - an unprecedented time when everyone was cooped up in their homes with very little outside activity. As work had shifted to remote and free time grew in my life, I decided to learn Python programming as it was gaining popularity in the automotive industry, specifically for data analysis. While learning about the basics of Python, one of the tutorials mentioned that it can be used for, among other things, web development as well. If I created a website using Python, my thought process went, then I would not just learn the basics of the language but also create a non-trivial project that would help Innocelf thrive. Besides it would give me a glimpse of work different than what I was doing in automotive. This seemed more appropriate to fill the time than watching television or playing video games, neither of which I owned at the time.

What began was a five to six month project of creating Innocelf’s website and client admin dashboard. Little did I know that there would be other hurdles lurking along the way named HTML, JavaScript and CSS. However, they are not the hero of this story but are merely cameos who played their parts in the development process. It all started with this:

Initial development (2021)
#

I chose Django to create the website and dashboard because in 2021, I did not know any better. Cursory searches on the internet like “python web development” praised Flask and Django, except Django was called a “full framework” with authentication, templating and other “batteries included”. I did not know what any of those terms meant or what a framework was, and naively thought that it was part of Python. After a couple of long YouTube tutorials (the longer than 5 hours type where they explain the fundamentals and walk you through a real project), I was up and running with the initial skeleton of a website. These tutorials also explained the difference between a backend and the frontend, and the technologies used in the both.

Django offered a great templating engine used for rendering dynamic content on web pages. I used them for simple things like {{ title }} of the page or the {{ description }} in the HTML head element. I also found myself copying a lot of the same HTML around to maintain consistency on different pages. This violated D.R.Y. (don’t repeat yourself) which is important in creating maintainable code. Vanilla JavaScript offered great ways to make the pages dynamic while being D.R.Y.

When you are new at something, you want to use the technology to its “fullest” potential. Similar to a novice musician who plays too many notes to “show their expertise”, I went completely overboard with the use of JavaScript in the name of “dynamic”. Creating elaborate classes and using extensive inheritance in very long JS files (> 6000 lines), for the perfect system. I was very close to creating my own JS framework (just kidding, it would have been terrible). Take a look at the a typical HTML file; its resemblance with Reach is uncanny.

<!DOCTYPE html>
<html lang="en">
	{% extends 'base_ca.html' %}
	{% load static %}

	<head>
		<meta charset="UTF-8">
		<meta name="viewport" content="width=device-width, initial-scale=1.0">
	</head>

	<body class="creative-lp">
		{% block content %}
		<script src="{% static 'js/chart.min.js' %}"></script>

		{% csrf_token %}
		<div id="app"></div>

		<script type="module"
			src="{% static 'ClientAdmin/js/action.js' %}"></script>

		{% endblock content %}
	</body>
</html>

What follows is the tree structure of the JavaScript files for the website landing pages. There are 9455 lines of JavaScript code in this one folder alone ¹.

.
├── actions_ts.js
├── ComponentClasses
│   ├── AnchorLinks.js
│   ├── CheckboxWithLabel.js
│   ├── ContactUsForm.js
│   ├── DropdownMenu.js
│   ├── DynamicTypedHeading.js
│   ├── FAQ.js
│   ├── FirstPageListItem.js
│   ├── Footer.js
│   ├── HeadingOrParagraph.js
│   ├── Navbar.js
│   ├── OneProcess.js
│   ├── OneProcessGapContainer.js
│   ├── SelectInputWithLabel.js
│   ├── ServiceDescription.js
│   ├── Testimonial.js
│   ├── TextInputCharacterCount.js
│   ├── TextInputWithLabel.js
│   ├── Tooltip.js
│   ├── TwoColumnContainer.js
│   ├── TypicalFormSubmitButton.js
│   ├── TypicalModal.js
│   └── TypicalPostForm.js
├── components_ts.js
├── footer.js
├── render_blog_home.js
├── render_privacy_disclaimer_terms.js
├── render_ts.js
├── renderAboutUs.js
├── renderBlog.js
├── renderFAQ.js
├── renderHomepage.js
├── renderServices.js
├── renderTestimonials.js
└── utils.js

2 directories, 35 files

Similarly, the tree structure of the JavaScript files for the client admin is shown below. 5051 lines of JavaScript code.

.
├── _download_invoice.js
├── _generate_invoice.js
├── action.js
├── client_admin_page.js
├── components.js
├── rend_inven_disc_quest.js
└── render.js

1 directory, 7 files

Although I tried to structure the code as experts would do, the result was, inevitably, a big bowl of mediocrity. It was horribly over-engineered for no apparent reason. It was brittle. Similar to an elephant balancing on an upright walking stick: we all know it is going to break, we just don’t know when. After the website and dashboard had been up for two years, we hired a SEO consultant to improve its search ranking. He pointed out how rendering everything with JavaScript was hurting SEO because JS files are loaded after the HTML. Search robots scan the page at its first load and seldom wait for the dynamic content to load. In the meantime, Innocelf was growing and there was a need for new features/automation in the client admin dashboard.

Technical Debt
#

JavaScript
#

In lieu of developing these new features, I opened the project after two years of initial development. Developers always think that their old code is worse than what they write now. In my case, that feeling was ten times worse. “I did not understand my own code” would be an understatement. I could not wrap my head around the complex structure of the code, and similar to most novice developers, there was no documentation. I had forgotten what each class did and what it was inherited from this other class; it was spaghetti code, no, it was rotten spaghetti code. The complexities of the frontend seeped into the backend. For example, there were multiple views to retrieve Django forms, i.e. empty forms, render them to the frontend, and logic to submit the form to the appropriate URL.

# views.py

def obtain_long_term_client_form(request, *args, **kwargs):
    '''
    The function gathers the long term client form and sends it to the frontend
    via XML request
    '''
    long_term_client_form = LongTermClientForm()
    return HttpResponse(long_term_client_form)

// Corresponding JavaScript

/**
 * The function obtains the long term client form from the backend and renders it nicely in a div
 * @returns Promise with the long term client form
 */
export async function _longTermClientFormRender() {
    let longTermClientForm = new ComponentServices.TypicalPostForm(
        'add-long-term-client-form'
    ).result;

    let formStringData = await RenderServices._obtainForm(
        '/client-admin/obtain-long-term-client-form'
    );
    let form = new DOMParser().parseFromString(formStringData, 'text/html');
    // let csrfToken = document.querySelector('[name="csrfmiddlewaretoken"]');
    let fullName = new ComponentServices.TextInputWithLabel(
        'Name*',
        form.querySelector('[name="client_name"]')
    ).render().result;
    let company = new ComponentServices.TextInputWithLabel(
        'Company*',
        form.querySelector('[name="client_company"]')
    ).render().result;
    let email = new ComponentServices.TextInputWithLabel(
        'Email*',
        form.querySelector('[name="client_email"]')
    ).render().result;
    let submitButton = new ComponentServices.TypicalFormSubmitButton('Submit')
        .result;

    longTermClientForm.append(
        // csrfToken,
        fullName,
        company,
        email,
        submitButton
    );

    submitButton.onclick = function (event) {
        if (longTermClientForm.checkValidity()) {
            event.preventDefault();
            let csrfToken = document.querySelector(
                '[name="csrfmiddlewaretoken"]'
            );
            longTermClientForm.append(csrfToken);
            _longTermClientFormSubmit(longTermClientForm);
        }
    };

    return longTermClientForm;
}

Postgres
#

A Django project is divided into apps. When creating the client admin app for the project, I used Pascal casing to name it: “ClientAdmin”. When Django migrations were run using this app name, it created Postgres tables with the following structure: “ClientAdmin_modelname”. Seems rather innocuous, and it would be if you were interacting with the database using Django. However, because Postgres reduces all upper case characters to lower case, it requires a double quote around the table name when working in psql.

The query SELECT COUNT(*) FROM ClientAdmin_project; leads to the following error, stating that clientadmin_project is not a real table.

To get around this error, one has to put double quotes around the table name. Therefore, SELECT COUNT(*) FROM "ClientAdmin_project"; works as intended.

This was not a problem when engaging with the tables via Django. However, it started to annoy me when trying to study data using psql. More importantly, it was inconsistent with the naming convention of other apps in the project, which all used snake case naming.

Rewrite
#

As I could not grapple with the existing code without breaking multiple parts of the project, I decided I would just rewrite the entire project while maintaining the existing Django app structure, but reducing JavaScript significantly. It was actually easier than I had previously anticipated. There were a few reasons for this:

Rewriting with context is like a green field project but with a set end goal.
Previous development allowed me to take a system level design approach. I was able to see where all the different pieces fit, and how to fit them effectively.
Over the two years, I had worked on two more web projects using FastAPI and Django. I, therefore, had more experience with the Django framework and its effective usage.
I had used Python for my automotive work as well, which made me more experienced and nuanced about its usage. Do less, achieve more.
I had more time. My company was going through bankruptcy proceedings, and no new development was happening on my day job.

Landing pages
#

The landing pages were the simplest to rework and reduce their heavy JavaScript dependence. After rendering the pages in any browser, I copied the outer HTML element for the div of id app using developer tools to paste them in the template file. Although there is still some JavaScript in the codebase, the number of files and lines has reduced significantly.

As seen from the new file structure for the JavaScript files, the number of files has reduced to eleven files (from thirty-five). There are now 1339 lines of code as compared to the previous 9455. This has improved SEO significantly and was indexed by search engines effectively. Most importantly, it has reduced the confusing code structure that only me from 2021 could understand, making it more maintainable and robust to upcoming changes.

.
├── ComponentClasses
│   ├── AnchorLinks.js
│   ├── DynamicTypedHeading.js
│   └── HeadingOrParagraph.js
├── render_blog_home.js
├── render_privacy_disclaimer_terms.js
├── renderAboutUs.js
├── renderBlog.js
├── renderFAQ.js
├── renderHomepage.js
├── renderTestimonials.js
└── utils.js

2 directories, 11 files

JavaScript that was preserved is related to dynamic typing and testimonials carousel of the homepage, and testimonials animation of the testimonials page. The rest was removed and clever classes from TailWindCSS were used to obtain the desired effects.

Remainder files contain functions to render SVG elements where required or do some minor automation tasks. For example, in render_privacy_disclaimer_terms.js, a SVG element is inserted in HTML using JavaScript.

// render_privacy_disclaimer_terms.js

'use strict';

import { _grayCirclesOfBlueBeltBackground } from './utils.js';

/**
 * The function renders the Privacy Policy Page /privacy-policy
 */
export function renderNavbarAndBrandDiv() {
    let brand = document.getElementById('brand-div');
    _grayCirclesOfBlueBeltBackground().then((response) => {
        brand.append(response);
    });
}

renderNavbarAndBrandDiv();

Client Admin
#

The rewrite of the client admin involved more work. Although the previous dashboard was functional and had relevant features, it was clunky and unintuitive. Rendering of some elements was absurd, i.e., they were either very large or very small, without the option to resize. This was a CSS issue, of course, however, it was very hard to update classes in the large JavaScript codebase. Moreover, similar functionality was not grouped together for effective and intuitive use. For example, adding projects was grouped together with creating invoices; two uncommon processes. Lastly, in my opinion, a dashboard should display key performance indicators of the system at first glance. For Innocelf, those were ongoing projects, revenue, and approaching deadlines.

The dashboard refresh included three plots on one side of the client admin’s homepage: revenue per month, revenue per project type, and year-over-year revenue of the company since it’s inception. The first two plots have a dropdown menu with it that lets the user choose the year for the plot. On the other side of the homepage is important project related information: near-term due projects, on-going projects, projects that are completed and are pending invoices.

Lastly, as a final intuitive push, a unified navigation is created to group similar functionality together. Each option within the navigation houses sub-options that are relevant to their parent.

Due to confidentiality reasons, the two dashboards cannot be shown and compared pictorially.

Projects
#

Project management is a large part of the client admin dashboard. At any given time, there could be multiple projects in progress and assigned to various individuals within the company. Projects also have payments associated with them to determine if the invoiced project was paid in full. This is particularly important for a large project that is divided into several parts, and payments are initiated when relevant parts are completed.

To render multiple types of projects depending on whether they are ongoing, completed but unpaid or cancelled, one projects.html file is used as a template in the new dashboard. The supporting view uses Django filtering to obtain projects depending on URL parameters. This allowed to encompass the same look for all projects pages on the client admin, regardless of their current state, and manage actions related to them consistently. The view also allowed to search projects using certain keywords, which are passed as a query to the get_projects function.

# views.py
def projects(request: HttpRequest, status: str) -> HttpResponse:
    query = request.GET.get("query") if "query" in request.GET else ""

    if request.method == "POST":
        search_term = request.POST.get("query")
        query = search_term

    is_asc = False
    context = get_projects(request, status, is_asc, query)
    return render(request, "ca/projects.html", context)


# urls.py
path("projects/<str:status>", projects, name="projects")

Each project row has options to perform multiple actions on that project. These actions are shown below. Of course, there are tooltips associated with each of these icons which activate on hover, so that the user knows exactly which action they are performing. A singular projects.html file and using Django templates allowed me to perform these actions seamlessly by using multiple views and a tags with an appropriate href. For example, the HTML template code example for marking a project complete is shown below.

<a id="{{ prj.uuid }}-project-status"
    href="/ca/mark-prj-comp/{{ prj.uuid }}?next={{ request.get_full_path }}"
    {% if project.is_project_complete %}
    class="relative group text-green-500"
    {% else %}
    class="relative group"
    {% endif %}
    data-popover-target="{{ prj.uuid }}-popover-project-complete">
        <i class="fas fa-check-circle fa-lg"></i>
        <div data-popover
            id="{{ prj.uuid }}-popover-project-complete"
            class="flex w-max px-3 absolute -right-6 bg-black text-white text-center lato-regular opacity-0 group-hover:opacity-80">
                <span>Mark Project Complete</span>
        </div>
</a>

In the previous implementation, this simple process was very convoluted and hard to accomplish. Any new project action would have taken similar effort to develop. For example, the previous implementation of marking a project complete is shown below. Even though the function below can perform two actions in itself, the code is much too verbose and was highly coupled with a class.

markProjectCompleteOrInvoiceSent(link) {
    let csrftoken = document.getElementsByName('csrfmiddlewaretoken')[0]
        .value;
    let packetToBeSent = {
        _elementId: this.result.id,
    };

    let xhttp = new XMLHttpRequest();
    xhttp.onload = (data) => {
        if (xhttp.responseText === 'Success') {
            if (link === 'mark-prj-comp') {
                this.markProjectCompleteButton.classList.replace(
                    'text-gray-500',
                    'text-green-500'
                );
            }

            if (link === 'mark-inv-sent') {
                this.markInvoiceSentButton.classList.replace(
                    'text-gray-500',
                    'text-pink-600'
                );
            }
        }
    };
    xhttp.open('POST', '/ca/' + link);
    xhttp.setRequestHeader('X-CSRFToken', csrftoken);
    xhttp.setRequestHeader(
        'Content-Type',
        'application/json; charset=UTF-8'
    );
    xhttp.send(JSON.stringify(packetToBeSent));
}

Patent Search Reports
#

At Innocelf, patentability search is one of the cornerstone services. Patent searches are unique for each invention, and the process of performing a search requires years of experience. However, Innocelf’s patent search reports follow a streamlined format homed in over the years. Drafting patent reports by hand comprises of about 40% of an entire patent search project. Non-trivial amount of time. I was trying to find ways to potentially reduce the drafting time, potentially eliminate it, by automating report generation.

What followed was a one month long development cycle to understand the patent search report and automate it. I could not eliminate all of the drafting time, but was able to optimize half of the quoted 40% used for drafting. This was a major boost in productivity for everyone working on these projects. This functionality was added as part of the client admin dashboard during this rewrite. In this functionality, the user has to input relevant patent identifiers, select information from those patents to be displayed in the report, and click Generate. A report is generated and downloaded to the user’s computer within seconds.

As Innocelf grows and our employee count increases, a patent report generator will allow for consistency across the company. This will not only allow us to focus on maintaining high quality of services Innocelf’s clientele are accustomed to, but also be more efficient in delivery.

Other Redundancies
#

In the initial development of the project, I had not set up database backups or full virtual computer backups. I was ignorant to loss of client information and/or project history if something were to happen to the VPS itself or the database. As business grew and number of projects increased, I setup a database backup system. It is currently a simple rclone to S3 buckets after a simple pg_dump.

pg_dump --no-password -U pm -d inno > /home/pm/innocelf_pgdump.sql
rclone sync /home/pm/innocelf_pgdump.sql InnoBDB:inno-db

I used find to calculate lines of code in JS files:
find . -type f -name '*.js' -exec cat {} \; | wc -l ↩︎

Background #

Initial development (2021) #

Technical Debt #

JavaScript #

Postgres #

Rewrite #

Landing pages #

Client Admin #

Projects #

Patent Search Reports #

Other Redundancies #