Not long ago I built this site that lets users rate images and provides a ranking. Yes, this looks similar to Zuckerberg’s notorious rank female faces site, but the project simply allows you to rank whatever images. The content is not my concern.
Backend
MongoDB was chosen as the database for storing user info, images and ratings. When my friends and I came up with the idea of this site, we also wanted to compare different aesthetic preferences among our friend circle, so we decided to pre-generate a queue of image pairs for all users to rate, and the progress of each user is stored as a field in the user model. The pairs are stored as a stand alone collection.
User model:
Queue model:
Rating and Image models are also created, but the structures are simple enough that I have no need to put them here. Rating model saves which user voted for which candidate in a Queue model; Image model just saves the image data and file type. A vote property is added to the Image model to simplify the sorting process.
The api endpoint design is also straightforward. There are three sub routers:
image
GET api for images
rank
GET for the ranking for all
rate
GET the next pair to rate with a user id
POST the rating of the user
There is no user login/signup. Every user is “invited” after the user information is put directly into the database. They can put in their unique Object id on the page to access the rating page. The ranking result page is public to all.
Frontend
The frontend is also very simple: A React Router handles a home page, a rating page, and a ranking result page.
The home page takes the user’s unique id and redirect to the rating page:
Today I did two things for my blog project: added a proxy on my Nginx server for the api connection and mounted /data/db directory from the host to the docker container to achieve data persistency.
First, Nginx proxy.
The idea/goal isn’t that complicated. There are three docker containers running on my production machine for the blog, by service names:
blog-api, the api server that listens to :1717
web, the frontend nginx server that listens to :80
mongodb, the mongodb database that listens to :27017
Before today, I had to configure the port in the frontend code, so that the frontend calls the api endpoints with the base url and the port number. If this didn’t bother me enough, for the image uploading, all of the urls for embedding the images in the posts have the port numbers in them, so they look like this:
vcm-3422.vm.duke.edu:1717/uploads/image.png
It is against intuition for the port number to be shown to users, so I began looking for a solution. Nginx turns out to have this reverse proxy configuration that allows you to proxy requests to some location to some other port number, or even a remote server. It’s called “reverse” proxy because unlike “normal” proxies, the nginx server is the “first server” that the client connects to, whereas in other proxies the “proxy server” is the first and the nginx server would be behind the proxy.
The upstream server name following the upstream keyword can be arbitrary, but the server name blog-api matches with my docker service name and will serve as the host name for the second hop as shown in the proxy_pass field below.
The location/api block does the proxying. The first line rewrites the request so that the /api part of the url is stripped since “api” is only used for triaging. The next few lines are pretty standard. They basically send along the original headers.
Voila! When I built and up’d my docker-compose services, I can see the blog posts showing up just like before. However, when I randomly tested image upload, I got a Request Entity too Large 413 error in the browser console. Apparently this is caused by the new nginx config, but how?
After some Googling, it turns out that nginx has a setting in HTTP server called client_max_body_size, which defaults to 1M. What’s more, from the documentation it says any request with body larger than this limit will get a 413 error, and this error cannot be properly displayed by browsers! Okay… so in the server block I added a
client_max_body_size 8m;
and everything works just fine 👌.
For the docker volume configuration, I came up with this requirement for myself because my mongodb data had been stored in the docker container – it gets lost when the container is removed. To back up the data easier and to have more confidence in the data persistency, I wanted to mount the database content from the host machine instead.
So in my docker-compose.yml,I added this simple thing to the mongodb service:
volumes:-/data/db:/data/db
Yes two lines and I included in my post. What can you do about it 🤨
Now I can just log in my production machine and copy away the /data/db directory to back up my precious blog posts data 🙂
Today I have been working on this feature that allows users to upload image as attachments and to include them as Markdown format.
Here’s a demo of the complete feature:
The first decision I made was how I should accept the upload. After several minutes of thinking with my pea brain, I decided to use multer to take file upload from clients. multer will put the file upload in a directory with a randomly generated name to avoid name duplications. It works well with express js in that it sets the ‘file’ property on req with useful properties such as filename, filesize and the path to the file.
Without much further thinking (which later proved to be a mistake), I thought it would be natural to simply serve the directory that has all the uploaded files.
This worked fine. However, when I deployed some other minor changes such as progress bar for upload and one click copy path button and hit refresh on my browser – the images were gone! I soon realized that it was because Docker created new containers because of the file changes, and the files on the original containers would be lost unless I do some backup or mount to the host.
That was when I decided to store all of the image files in MongoDB. In this way, the images are staying at the same place with the post contents, which makes backing up easy. It would also be easy to implement because I already had code for other schemas.
I had to also catch ‘CastError’ in GET because the stupid mongoose throws when the param cannot be casted into ObjectId.
Basically what I do when the user uploads is storing the file to MongoDB, deleting the file in FS, and returning the ID of the file in the database. The ID can then be used for the GET api.
I’m also proud to say that this API endpoint is unit tested and (almost) fully covered. No I wish not to discuss the overall 0.2% coverage drop for the repo.
As I said above, I also added progress bar, feedback for copying path, error prompt for invalid files and some other UI features. The GitHub issue is now closed and I now just have to wait for the requester to come back online for my demo :).
All right, this is where I say I can actually get something done.
Achievements for the blog project include:
APIs for log in/out, posts CRUD, comments CRUD, like/dislike
93% coverage on APIs mentioned above
Using React-Redux to maximize data reuse and minimize the number of API calls
Using universal-cookie to store the logged in state (okay this might not deserve a stand alone bullet point)
Using Docker (Dockerfile and docker-compose) to automate the deployment process.
Today, lucky for you, I’ve decided to talk about how docker-compose in this project works.
Docker is the company driving the container movement and the only container platform provider to address every application across the hybrid cloud.
^ From Docker’s self introduction. What that means for me is that with proper usage, I wouldn’t have to set up production machines with all the dependencies that my project needs whenever I would like to deploy. Ideally all I would have to do is to write Dockerfiles and docker-compose.yml, install Docker and let Docker handle the rest.
In this blog project, separating the backend and the frontend, the dependencies (required on the environment, not the npm ones) are:
backend:
MongoDB
Node/npm
frontend:
Node/npm (for building)
Nginx (for serving)
With these in mind, I was able to write a Dockerfile and a docker-compose.yml for the backend following documentations and random StackOverflow answers online:
Dockerfile:
FROM node:carbon
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build-server
EXPOSE 1717
RUN ["chmod", "+x", "/app/wait-for-it.sh"]
CMD ["node", "build/server.js"]
The Dockerfile specifies the config for the blog-api container, while the docker-compose.yml tells Docker how my blog-api container relates to the mongodb service container.
Several things to notice:
Each Docker container is like a VM by itself, so the WORKDIR is the directory in the container, and when I do a ‘COPY . .’, naturally it copies from the current directory in the host to the current directory in the container.
Notice how I copied the package.json file first and npm installed before copying anything else. The reason for this is that Docker uses a layering cache system that is able to reuse previous versions of images if nothing changes in Dockerfile. Therfore if I only change some api route file, I wouldn’t have to wait for the long npm install process again.
wait-for-it is a tool to wait for a process to listen to a port before doing something. It has automatic retires that is very useful in this case. I could, however, just let blog-api restart always as is, but this tool doesn’t have as much overhead.
Later I added another Dockerfile for the frontend, which looks like this:
FROM nginx
RUN apt-get update
RUN apt-get install -y curl wget gnupg
RUN curl -sL https://deb.nodesource.com/setup_8.x | bash
RUN apt-get install -y nodejs
WORKDIR /app
COPY package*.json ./
RUN npm install
COPY . .
RUN npm run build
RUN cp -a /app/dist/* /usr/share/nginx/html
RUN cp /app/nginx.conf /etc/nginx/
This image extends from nginx, so the default CMD starts up the nginx server. I need nodejs for building the static files, so I added the couple lines there. The last two lines copy the static files to nginx’s serving directory and my config file to nginx’s config directory.
With the frontend added, I added one more service to docker-compose.yml:
This simply links my container for the web frontend to docker-compose so that I wouldn’t have to manually start up every container. Instead, I would only have to do docker-compose build and docker-compose up -d.
I also added automatic seeding for the MongoDB database but I’m too lazy to paste the steps here again so screw you.
This following point is unrelated to Docker, but I spent some time on it and felt like it would be interesting to include here. It is my nginx.conf file. Since I’m building the frontend with React single-page-serves-it-all pattern, I have to make sure that the nginx server returns the index.html file no matter what the sub url paths are. The only exception is that the client is requesting some js or resource file. With this in mind:
server {listen80;
root /usr/share/nginx/html;
location /{
try_files $uri /index.html;}}
It tries to file the file specified in the uri first, before returning index.html regardless. 404 is handled on the frontend by my React application.
For the next step, I’ll be working on attachments to posts as a feature request from this person.