Licence

Online Judge - RCE Engine

Online Judge is an RCE Engine for coding platform like LeetCode, Hackerrank.

Online Judge can execute C++, Python and Java code in secure container, compare code output with testcases and return appropriate response to the client.

General Information

Online Judge is an experiment to implement the Judge with AlgoCode Platform a DSA platfrom just like Leetcode.

AlgoCode is a coding platfrom built in microservices architechture. To learn more about AlgoCode, please explore this repository: AlgoCode Backend

Online Judge can execute user codes and run test cases against the code output.

The Judge can handle the below events:

a. AC (Accepted) 
b. WA (Wrong Answer)
c. Compilation Error 
d. Time Limit Exceed 
e. Memory Limit Exceed 
f. Segmentation Fault

The Online Judge is a pure docker implementation and no other 3rd party API or service has been used.

Run Locally

Clone the project

  https://github.com/Mahboob-A/online-judge.git

Go to the project directory

  cd src 

Create a virtual environment

For Linux/MacOS

  python3 -m venv venv

For Windows

  python -m venv venv

Activate the virtual environment

For Linux/MacOS

  . venv/bin/activate

For Windows

  .\env\Scripts\activate.bat

Install the dependencies

  python -r requirements/dev.txt 

Create the docker image

You must creat the below image and volume in order to run the Judge successfully.

Go the the path to Dockerfile

  cd docker/lang/cpp

Run the command to create the required images

  docker build -t algocode/revamped-cpp:v1 . 

Make sure to add the "dot" at the end of the line.

Create the docker volume

The main Judge Container and the Sibling Containers use a Volume to share data between them.

Run the command to create the desired volume:

  docker volume create user_code_files

You need to create the volume as this volume will be managed externally.

Run the Judge

Return to the src directory, and then run the bellow command to start the Judge. It will also run the development server.

For Linux/MacOS

  make docker-up 

For Windows

  docker compose -f dev.yml up --build -d --remove-orphans

Now you are ready to send request to the API asper API reference below. Wait for 5 seconds to propage the Judge before sending a request.

Stop the Judge

To stop the Judge, run the below command.

For Linux/MacOS

  make docker-down

For Windows

  docker compose -f dev.yml down

API Reference

Submit Code for Execution

  POST /submit/execute-robust/

Parameter	Type	Description
lang	string	Required. The language of the Code. Currently only cpp is available.
code	string	Required. The C++ code you want to execute.
input	list[string]	Input for the program. Empty if no input is neeed.
testcase	list[string]	Testcase that should be compared to check th e answer of the code execution.

All the code are pushed to the project as PR. Hence, you can view the work has been done in the closed PR of the project. Detailed explanation has been added to understand the PR. To learn more on API, please take a look at this PR: Judge API

Judge Without API Call

If you are just curious to test it locally without calling any API, you can also do that.

Follow the below steps -

A. Clone This repository: Judge without API Call

B. A step by step guideline has been outlined in the below mentioned PR, please follow along the PR to run the Judge without calling the API: See the PR. It's really well documented, I promise!

Troubleshoot

If the Judge is not responsive, you can stop and restart the Judge using the commands mentioned in the Run Locally section.

If this didn't help, stop the Judge, go to src/core_apps/judge_engine/containers.py, add a print statement on logs and status_code after line number 170 and 171.

Run the Judge, and notedown the container ID using: docker ps.

Run docker logs -f <container ID>

And you will be able to see the Judge container's development server's output in your terminal.

FAQ

What is the backend for this project

The backend of this project is implemented with Django.

Language

Python

Why testcases and input are being passed from client

This is a Revamped Implementation of Online Judge for AlgoCode - A DSA problem solving platform implemented in microservices architecture just like Leetcode.

This Judge simplifies to the code execution. If you just want a monolithic application to test how an secure code execution should happend with test cases comparison to generate a DSA platform like answer, this is a handy tool.

This Judge is implemented in the Remote Code Execution Engine Service of the AlgoCode Platform.

How secure is the execution

The C++ code will be executed in a secure docker container. The container has no privileges and all other possible privileges has been dropped.

The container can handle any kind of DoS Attack like fork bomb and any other form of resource starvation attack.

A new sibling container is spawn to execute the code.

As the container is auto terminated within 5 seconds, and only allocated .5 cpu, 300mb of RAM with no swap available, hard limit of nproc is set to 50 and hard limit of nofile set to 50, it can not run any heavily process or descriptor oriented tasks.

However, you can tweak the settings as per you need as well.

Why only C++ can be executed

The main AlgoCode platform is able to rub Python, C++ and Java code. However, This RCE engine will also be updated as I continiously develope the AlgoCode.

In future updtes, Python and Java execution will be added.

If you need immediate support of these languages, please do not hesitate to connect with me at: [email protected]

Future Improvements

Spawning new container and stop it and cleaning it is expensive process. Hence I am planning to optimize this use case and if the below are achieable, I will bring them in my future rolling of the Juddging engine.

Explore the BOTEK Estimation

As creating new contianers and deleting them are expensive process, maintain the container's state and execute the code in the contianer without stopping or closing them.

As the image of the container is very lightweight, we might be able to run concurrent 100+ containers with 64 GB of RAM, and 2 Core CPU.

But the overhead here is not the number of containers, the overhead is networking and the bigger overhead is disk seek, as the judge needs to write the necessary files to the disk in order to mount it with the container, and again, delete them after the execution is successful.

The Optimation

If we constantly run 100+ contianers as per our above estimation, and we mount the user files directory dynamically before the code is executed, we can really mitigate almost 1.5/1.0 seconds of cold start of the docker container.

In this design, we can increase the speed and efficiency of a Synchronous backend like Django and it would able to handle more concurrent requests with less time.

My Plan

Yes, I will implement this in my next release. In the future release, the Online Judge RCE Engine will be faster and mitigate the cold start of the docker container.

Screenshots

API requst in postman

Wrong Answer Response

Logging Level

The architechture of AlgoCode Platform.

🔗 Links