What was the challenge for me?

I'm mostly an introvert person and to speak up in front of even 10 people is daunting for me. Participating in a hackathon where I'll need to brainstorm ideas with unknown folks would be a bit outside my comfort zone. I had participated in hackathons during my undergrad days but had mostly worked alongside my college friends. Nonetheless, I channeled my inner coder strength to convince myself to participate and meet some tech-savvy women in the process. Yep, that was the motivation! :)

What was wmn;2023?

It was an onsite women only 24 hour long hackathon organised by Devfolio.

Ideation Phase

I was working at an insure-tech company back then and had some understanding of problems faced by Indian insurance companies and the gaps in customer expectations and insurance limitations while I was very impressed with the insurance systems in first world countries like Singapore. I thought of building a project on a similar idea. At the hackathon, I met a girl working in healthcare and we brainstormed our ideas for the prototype.

What did we build?

The idea was to solve the overall claims experience at the provider end i.e. insurance providers to be able to detect fraudulent claims and reject them as well as ensure an entire cashless flow from the customer's end without the manual intervention for customers to preserve the receipts from the consultation/treatment and uploading it later to get a reimbursement.

How was it solving the provider experience?

One of the major pain point of an insurance provider is to be able to detect a fraudulent claim. Especially in OPDs with low ticket size, the possibility of fraud increases even more and that is one of the reasons why insurance providers in India don't go for OPD included in the benefits. Another problem is data not being shared by different providers leads to a chance of duplicate claims.

How was it solving customers' experience?

OPD coverage - Most insurance providers in India do not provide OPD coverage. However, from the customers' point of view, this is the biggest economic cost center in health. With lesser possibility of frauds, insurance providers will be able to offer more holistic coverage including OPDs.

Cashless experience - Most of the insurances have a reimbursement flow and no coverage for OPD which leads to saving all the receipts, uploading them, and waiting for weeks for it to be approved.

Privacy of PII data - If there is a central system for storing all the health records, as a customer I should be able to control that data access. Each time an insurer requests my data, I should be able to control what document data I want to share, and only upon my authorization should that data be shared.

Demo link

Adding a demo link for the project -
Video - https://www.loom.com/share/5d235242d6f14470ba1bbcb7830a8c96?sid=48fdd704-788b-4b80-82b4-d1cb63490b72
Project link - https://devfolio.co/projects/insurease-50a3

Winning the grand prize

And yeah that was it. After a really long time, I pushed myself to stay up for ~30 hrs to finish the prototype project as a solo developer. After all, it was about pushing my boundaries. I saw the announcement on twitter the next day. :)

Origin and same origin policy

Origin is the source of a particular web request, where the request is originated or is being executed.

What qualifies as the same origin?

Two URLs have the same origin if they have the same

1. protocol

2. domain

3. port

   

Below URLs qualify as having the same origin as the above highlighted URL

1. http://www.google.com:443/maps

2. https://www.gooogle.com/foo

Below URLs do not qualify as the same origin as the above highlighted URL

1. http://www.google.com/maps - Different protocol

2. https://google.com/maps - Different domain

3. https://ww.google.com:8080/maps - Different port

Same origin policy

To prevent different sites from accessing each other’s data (cookies, web storage, run scripts), same-origin policy is enforced by web browsers, and its enforcement dates back to 1996. This policy controls interaction between two origins when using HTTP requests. If you try accessing another origin, you’ll get a CORS error. In modern web browsers, CORS is disabled by default.

Let’s try this out! ✨

First, we’ll open dev tools on any website. Currently, I’m on https://www.google.com and let’s also spawn a node web server.

const express = require('express')
const app = express()
const port = 3000

app.get('/', (req, res) => {
  res.send('Hello World!');
})

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

Now I have a NodeJS server running on port 3000.

We’ll make a simple fetch call in the browser console to get the response from our server.

fetch("<http://localhost:3000>").then(res => res.text()).then(x => console.log(x));

On executing this, we got an error like this

Access to fetch at '<http://localhost:3000/>' from origin '<https://www.google.com>' has been blocked by CORS policy:
No 'Access-Control-Allow-Origin' header is present on the requested resource. 
If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

and if we check the network tab, we see the following error

The error says we cannot access http://localhost:3000 as the origins mismatch and the server running at localhost should allow requests coming from the origin https://www.google.com in order for us to proceed.

Let’s make a few tweaks to our application code

const express = require('express')
const app = express()
const port = 3000

app.get('/', (req, res) => {
  res.setHeader('Access-Control-Allow-Origin', '<https://www.google.com>');
  res.send('Hello World!');
})

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

and try making the same simple fetch call in the browser console.

Voila! 💫 It worked. We got a response.

Promise {<pending>}
Hello World!

Note that we added a special header [Access-Control-Allow-Origin](<https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Access-Control-Allow-Origin>) to response headers. and we set its value to https://www.google.com. This is a way for the server to tell the browser that it trusts requests coming from [<https://www.gooogle.com>](<https://www.gooogle.como>) origin. We can also set the value of this header to “* ” wildcard which means it can allow requests from any origin and not just google.com.

Now let’s try something fancy! Let’s add a custom header to our simple fetch request

fetch("<http://localhost:3000>", {headers: {'a': 'b'}})
.then(res => res.text())
.then(x => console.log(x));

Oops! We’re again getting the CORS error

Access to fetch at '<http://localhost:3000/>' from origin '<https://www.google.com>' has been blocked by CORS policy: 
Response to preflight request doesn't pass access control check: 
No 'Access-Control-Allow-Origin' header is present on the requested resource. 
If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

An opaque response is a type of response that doesn't allow the web page to access the response's data, headers, or status.

Note that my emphasis was on simple fetch request. Read more about simple requests here. We made it fancy by adding a custom header to it. This simple request does not trigger a CORS preflight request. Oh! again a fancy term preflight request. Let’s get into that!

A preflight request is an OPTIONS request automatically made by the browser to check if the server is ready to accept the actual request, headers and allows the request from the given origin. This is a lightweight request and server usually responds with 204 No content if it agrees to the request conditions or can return an error. It is not triggered for simple requests.

Since our app.js does not have handling for that OPTIONS request, it is throwing the error. Let’s fix that

const express = require('express')
const app = express()
const port = 3000

app.get('/', (req, res) => {
  res.setHeader('Access-Control-Allow-Origin', '<https://www.google.com>');
  res.send('Hello World!');
})

app.options('/', (req, res) => {
  res.setHeader('Access-Control-Allow-Origin', '<https://www.google.com>');
  res.setHeader('Access-Control-Allow-Headers', 'a');
  res.sendStatus(204);
})

app.listen(port, () => {
  console.log(`Example app listening on port ${port}`)
})

Note that we also added a response header to allow the custom header without which we’d again be getting CORS error.

Now if we hit our fancy fetch request again, we’d notice that there are two requests that browser sent to the server. One was the OPTIONS request for which it got the below response

Here we can see that the server has allowed the origin and also the custom header.

and we got the 200 OK response for the GET request

Promise {<pending>}
Hello World!

How to block cross origin access

https://developer.mozilla.org/en-US/docs/Web/Security/Same-origin_policy#how_to_block_cross-origin_access

This is particularly useful in avoiding CSRF attacks.

How to allow cross origin access

• Using the Access-Control-Allow-Origin header

• Changing origins

  Some browsers provide the capability to change domain so that the requester can pose as the same domain and request can go through without the CORS error. However, this is limited to superdomains of the current domain only.

  For e.g. I can change document domain for https://foo.bar.com to https://bar.com by doing this on https://foo.bar.com

    document.domain = '<https://bar.com>'
  

  But I cannot change it to http://localhost:3000 [Not a superdomain]

  Google chrome has deprecated this citing security reasons, but Safari allows it.

  

• PostMessage

https://developer.mozilla.org/en-US/docs/Web/API/Window/postMessage

Examples

Let’s say a vulnerable social media website allows html content in an input field. An attacker can enter malicious html in that input field. This script which will be executed every time any user of the vulnerable website visits that page. Such type of attack can impact mass users with a single attack.

In this particular example whenever John or Alice load their timeline, the script authstealer.js will automatically run in their browsers and Evil will get access to their session data and compromised accounts. Evil can now login to John and Alice’s accounts and see their payment information, PII data etc.

Let’s try XSS! ✨

We’ll just create a simple NodeJS server, and a basic HTML and try injecting some JS into that

Here’s my index.html

<!DOCTYPE html>
    <div>Enter your name</div>
    <input type="text" id="name"></input>
    <button onclick="addData()">Submit</button>
    <script src="test.js" type="text/javascript"></script>
    <br/>
    <br/>
    <div id="self-xss"></div>
</html>

function addData() {
    const name = document.getElementById('name').value;
    const parsedDocument = document.createRange().createContextualFragment(name);
    document.getElementById('self-xss').appendChild(parsedDocument);
}

which renders a web page like this. This website has an input field that takes input names and displays them in a div.

Now, we’ll try doing something evil 😈. Instead of entering any text name, we’ll try injecting a script in the input field here

which is as simple as <script>alert("XSS attack")</script>

What happened here is instead of listing down the input, the script got executed here. And instead of that simple script, I can have the script downloaded from some other source and try to steal the credentials and session data of logged-in user here.

How to prevent XSS?

1. Safely Validating input fields - Mostly websites allow very limited set of HTML markup or do not allow HTML markup at all in input fields.

2. HTTP only cookie - This makes cookies unavailable to client side scripts.

3. Disable JavaScript in browsers. But that would make our website unresponsive. We’ll just be able to load static HTML pages. This might work for some use cases where we have just static HTML websites.

4. Send a request header 'X-XSS-PROTECTION' and its value can be set to none or some origin value stating that it will accept scripts only from a particular origin. Please note that this also disables inline scripting in html pages.

5. SameSite cookie parameter

Who is a trusted user?

A trusted user for a website can be an authenticated user with valid session data. The attack gets worse if the trusted user has administrative privileges to the website.

How it works?

The attacker poses as the trusted user (who is unaware of the ill intention of the attacker) to perform an action and it mostly tries to alter the state of website’s backend data. The attacker executes this action via social engineering e.g. a hidden link in email which when clicked executes the script or hidden link in img tag which does not even require user to click any link but will be executed as soon as the page is loaded. It may look like this

<img src='<https://yourbank.com?payeeId=evil&amount=10000>' height=0, width=0>

<a href='<https://yourbank.com?payeeId=evil&amount=10000>'> View Cute Pictures! </a>

or some POST request like this

<form action="<https://yourbank.com/transfer>" method="POST">

<input type="hidden" name="acct" value="EVIL"/>
<input type="hidden" name="amount" value="10000"/>
<input type="submit" value="View my pictures"/>

</form>

or PUT

<script>
function put() {
    var x = new XMLHttpRequest();
    x.open("PUT","<http://yourbank.com/transfer>",true);
    x.setRequestHeader("Content-Type", "application/json");
    x.send(JSON.stringify({"acct": "EVIL", "amount":10000})); 
}
</script>

<body onload="put()">

Web browsers by default send all the cookies and session data of the domain website when the request is made to that website. So when the request is executed by the trusted user (unknowingly) the vulnerable receiver website will validate the session data and allow the request to go through.

How to prevent CSRF?

1. Disabling CORS - Most modern browsers disable CORS by default which means cross origin requests won’t go through. However, attack can still happen if the target website has enabled CORS by putting * in 'Access-Control-Allow-Origin' header.

2. 'X-CSRF-TOKEN’ - Setting this token explicitly in headers. Backend server can generate a token (random and time based) and send it to the client in response cookies for preflight request. Client can parse the cookie and set the header while making a new request to backend server. This token should be a random and unique value with expiry for each request. However, this approach is based on the fact that the attacker script is unable to read the response cookies and set the header parameter.

3. Checking the referrer header (Origin/Referer header)- Backend server should always check if the referrer is a domain other than the one they intended to allow and block such requests from going to the application servers. Reliability on these headers comes from the fact that they cannot be altered programmatically as they fall under forbidden headers list, meaning that only the browser can set them.

4. SameSite cookie attribute - Setting this cookie value to Lax, Strict, or None . This attribute helps the browser decide whether to send cookies along with cross-site requests.

5. Do not use GET requests for state changing operations.

An ideal log may look like:

log(level: “error”, trace_id: “”, message: “Custom error message with context”, data: {“host-name”: “”, “host-ip”: “”, “contextual-key”: “contextual-value”}, timestamp: “”, stacktrace: “”)

Why a proper logging is needed?

1. Well, first and foremost we all make mistakes. Having a logging system with stack trace, log levels, contextual data, and errors (if any) can help us identify an issue in no time.

2. It is great for collaboration and having to work with multiple team members (unless you wanna remain the forever on-call person 😛). It’s helpful for a new person who gets onboarded to your service to understand the errors/context easily.

3. Logs help in tracking your user journey and identifying issues. (P.S. You can set up alerting and monitoring on top of your logs/some log patterns too. I have done it in the past and it was super helpful.)

Logs delivery (Where can you store your logs?)

• Console output — Very common and basic logging strategy for debugging. This is very simple and easy to use and handy when developing an application. Cons — logs won’t persist after the console is closed.

• File write — Your application can keep writing logs to a file, and so the log events will be stored on a local disk. And they’ll persist even if you close the console. However one issue that it may lead to is fast utilization of disk space on your application server.

• HTTP — You can make an HTTP call to your log service with the log payload and it’s up to the log service to process and store it the way it wants (well, the way it’s configured rather 😅). But this leads to a whole new set of issues, i.e delay in application response time. If you add too many logs to your application and it makes a synchronous HTTP call to log service every time, it’ll add a lot of delay in your application response time because of HTTP encryption/decryption/authentication/acknowledgment (and who doesn’t want a faster response time for their application).
  ▹ Solution — Make asynchronous calls. Yes, I’m coming to the downside for this too, don’t be judgemental so soon. Making asynchronous calls on a separate thread to the log service does solve one problem of delay in application response time. But, what about retries? What if your async HTTP call to log service fails and your application crashes too! Imagine having to debug the cause of the application crash when your error logging call failed too :(.
  You can go for a hybrid approach, wherein for the severe logs, you make sync calls to log service (ERROR level log for e.g.) and for debug/verbose you can make async calls.

• Queueing — You can push log messages to a queue and they can be processed later, in batches or as single events, like a Kafka streaming platform. You can build a retry mechanism on top of it too.

You may go for a combination of these depending upon the requirements. For e.g. can write logs to a local disk and other software reading that file at frequent intervals and push it to the log service (something like logRotate).

Some tips

• Add a uniform source ID especially if you’re working with microservices. What that means is you generate a request ID/UUID for a new request that comes from the client and you tag all the logs that are generated across different microservices throughout this request lifecycle with this unique id. This comes in handy when you want to track the user behavior/abnormality across microservices.

• Always add log severity very literally to the context of why you’ve added a particular log. There are 4 log levels, DEBUG, INFO, WARNING, and ERROR. One common mistake people make is adding everything to an INFO level log. Adding logs without the proper level may lead to chaos and you may not find the actual log that you’re looking for in times of crisis.

• Make sure to add timestamp at the application level to track the exact timestamp of the event. If you add timestamp of log entry, it may not respresent the actual timestamp of event as there could be some delay in log registration.

• Find a tool for centralizing your logs.
  ▹ If you’re working with distributed systems, you’ll likely have logs on different servers, which may be spread across different microservices, and fragmented. Having a centralized tool is very helpful for monitoring and troubleshooting issues in a certain time period.
  ▹ Look out for tools that allow you to filter, query, index, and search data according to your needs because you cannot always work with plain text logs.
  ▹ Check support for multiline logs. If you have stored stack traces in error logs and your logging software treats each line as a separate log, it becomes super confusing to look at.

• Don’t forget to remove business info-related logs while creating a release build.

Really, in the end, it should be you deciding what to use for your application. There’s no need to over-engineer your logging system (any software actually 😅)if your system doesn’t have the requirement for that.

Pros-

• Uses constant space (Space efficient).

• O(1) insertion and retrieval

Cons-

• False positives. It can not accurately tell if a key exists for sure in the bloom filter. But it can tell if a key does not exist in there.

How it works?

It requires some hash functions and a binary hash table.

First, we set all the bits in hash table to 0

Let’s say we have two hash functions H1 and H2

Insertion:-

→ Insert “Hello”. H1(“Hello”) = 0, H2(“Hello”) = 3

→ Since bits at 0 and 3 positions are set to 0, we toggle it to 1.

→ Now the hash table looks like this

→ Insert “World”. H1(“World”) = 5, H2(“World”) = 3

→ Toggle bits. Now the hash table looks like this

Searching:-

→ Search “World”. H1(“World”) = 5, H2(“World”) = 3.

→ Check bits at 3rd and 5th index. Both are set to 1. So it would return 1. Maybe exists case

→ Search “Foo”. H1(“Foo”) = 0, H2(“Foo”) = 5

→ Check bits at 0th and 5th index. Though it will return 1. But “Foo” does not exist. False positive case.

Applications:-

It is mostly used in conjugation with primary DB to check if an element is present or not and then search the primary DB. It can be used at places where we are okay with false positives and a quick lookup can save the expensive DB read.

• Used by Medium to show if an article is read by the user.

• Malicious sites blocker by Google.

• Password and username duplicate check.

• Used to show target ads.

Default support with databases

• Postgres — Postgres supports bloom index which is pretty useful for membership check cases. Link — https://www.postgresql.org/docs/current/bloom.html

• Redis — Redis (4.0 +) has ReBloom library which has bloom filter application. Link — https://redis.com/blog/bloom-filter/

• Some databases like HBase and Cassandra by default use bloom filter to filter out queries.

Advantages:-

1. Faster querying - Queries can be parallelly processed for range and aggregate queries and match queries can be worked up with smaller subsets instead of searching in bigger tables thus reducing query time.

2. Data Archival - By dividing data into smaller tables, less frequently used or unused partitions can be moved to cheaper storage thus making more room for recent data and optimizing on the costs.

3. Maintenance - We can easily run bulk update and drop partition queries without obtaining a global lock on the entire table blocking our read-write operations.

Disadvantages:-

1. Data updation across partitions - Data updation becomes tricky especially when updation leads to the movement of table rows from one partition to another. Let’s again take the example of our score table. If update marks for id = 411 from 51 to 98, the said row will need to be transferred from partition score_4060 to score_80100.

2. Searching via indexes - If the index key is other than the partition key, maintaining those indexes also becomes a challenge. This becomes more difficult when partitions are separated across nodes (sharding). We can though manage this by maintaining individual indexes at the partition level but if we’ve to search using the index, it will have to search all the indexes and then collate the data before returning to the application or the application will have to maintain this collation for some cases (e.g. inheritance partitioning in postgres). Another solution is to maintain the indexes at partitioned table level and each partition table can keep a copy of the same index.

Try it out ✨

Let’s try to create a partition table on Postgres database. Please note that the below experiments are being done on Postgres database and the results may vary for other databases.

For this experiment, we can use locally installed Postgres or a docker container. The goal here is to get a postgres server running. We’ll create a table named score having columns id and marks, and partition it by marks.

CREATE TABLE score (
    id SERIAL NOT NULL,
    marks INTEGER NOT NULL
) partition by range(marks);

and create some partitions

create table score_0020 partition of score for values from (0) to (20);
create table score_2040 partition of score for values from (20) to (40);
create table score_4060 partition of score for values from (40) to (60);
create table score_6080 partition of score for values from (60) to (80);
create table score_80100 partition of score for values from (80) to (100);

If we check our table is created now

Note that the partitioned_table score is just a virtual table with no storage to it. Storage actually belongs to the partition tables score_0020, score_2040 and so on.

Let’s seed our database table with some random data

INSERT INTO score(marks) 
SELECT floor(random()*100) 
FROM generate_series(0,10000000);

And we’re done!!

Let’s try out some queries

Let’s also try creating an index on column marks

CREATE INDEX marks_idx ON score(marks);

and check for the index status

Indexes automatically got created on partition tables as well. And the best part is our application does not need to worry about which partition to query from. PostgreSQL automatically propagates the index to each partition.

If we search for any marks, it went for searching only in the partition score_0020 and also utilized the index.

Let’s also create an unpartitioned table for benchmarking

create table score_unpartitioned(id serial not null, marks integer not null);
insert into score_unpartitioned(marks) select (random()*100) from generate_series(0,10000000);
create index score_unpartitioned_idx on score_unpartitioned(marks);

As we can clearly see, query is much faster with the partitioned table in comparison to an unpartitioned table as the query has to search just on the partitioned table.
An important point to note however is indexes, partitions and optimisation of queries depend on the size of data that we have and the acceptable response time for the queries that we have and of course, there is the cost factor associated with creating multiple indexes and partitioned tables. Number of partitions and strategy for partition also depends on the size of data and the type of queries that you have to run. YMMV!

Database indexes are used for faster data retrieval. Let's explore the underlying data structure behind these indexes that enables faster queries.

Hash Index

This is a log based index and probably simpler and easiest to implement. It stores data in key value pair format in an append only file. When the file size grows it just moves to storing data in a different file. Compaction happens in the background asynchronously. Compaction is the process of merging two files and using the latest one as the trusted and updated source of data.

SSTable

SSTable is Sorted string table. Similar to hash index, it also stores key-value pairs in an append only segment file but in a sorted order sorted by key. Compaction also happens in the background on a regular interval.

How does it maintain sorted order in a segment file?

It uses any sorted data structure like Red Black Tree or AVL tree to keep keys in sorted order.

How compaction works?

SSTable make writes faster by appending data in an append only segment file. On files reaching a certain limit, new writes are done on new segment file. Compaction takes values from latest segment file and rejects values from old segement files in case of repeated keys. Since SSTables are already sorted, merging is faster as well with help of any sorting algorithm like merge sort.

Searching

For searching any key in SSTable, we need two offsets to begin with. For e.g. if we’ve to search for offset value for key10 and we’ve got offset values for key 2 and key 15, we can just use binary search to search within the SSTable since the values inside the table are sorted by key. Generally a sparse in memory index is also kept which keeps offset values for random keys to help with the binary search process.

Deletion

Deletion is worth mentioning for LSM trees as this might be the only case where deletion actually takes extra space. Since no in place updates are allowed in segment files(append only files), if a key pair is deleted, we just add a tombstone value to the key name and while merging/compaction it reads the value as tombstone and does not carry it forward in the merged segment.

LSM Tree

LSM tree is log structured merge tree which is essentially the algorithm we just described earlier. Searching works in the same way as above, however worst case complexity is still high in LSM tree, going through sparse index and going all the way to all on-disk SSTables. Further optimizations in LSM tree are made by introducing a bloom filter. So search is first made in bloom filter, then it goes to LSM tree.

BTree

BTree is a balanced tree. It possesses the properties of a n-ary binary tree with a twist that it balances itself on every insert, delete, and update.

Each node in B-tree actually represents a page on disk. Contrast to LSM tree which divides disk into segments, B-tree splits and divides it’s data between nodes (pages) of fixed size. Insertions are slower than LSM tree but lookups are faster. B-tree also keep a WAL (Write Ahead Logs) which is an append only log file for data recovery in case of crash e.g. failure to update/add a value in the tree due to server restart or network failures. B-tree can again be reconstructed using that WAL.

B+ Tree

This is just an improved version of B tree in a sense that instead of storing references right next to the keys making the nodes heavier, it stores all the references at the leaf node. Makes range queries much faster.

Other indexes

Clustered index - Some indexes prefer keeping actual all row data in the value for key while some prefer keeping references to actual row data in the key while indexing data, the former one is known as clustered index.

Covering index - It is a mid-way between clustered and non clustered index. It stores values for some cloumns of the row and also the reference to original row. This allows some queries to be answered just by looking at the index and for more data, we can refer to the actual values.

Multi-column indexes - This is required for querying multiple columns of a table. E.g. if we’ve to filter out restaurants that lie within a given lat, lon values.

Most common multi column index is concatenated index, R-tree is also one of the multi column index.

Column oriented storage

→ In case of repetitive data in columns, mostly occurring due to joining tables or merging data from different sources e.g. in case of analytics data table, there is a lot of repetitive data if we just looks at the columns. Some analytics DB, store columns seperately in another table and just keep references in the main table. Further compression can be done on columns e.g. in case of number values, sparse values can be represented in a better way to save storage.