[
Smart way to Capture Jobs and Process Meta Data Using DynamoDB | Project Demo | Python Templates | Alerts
In this article, I will present a solution that will allow you to easily monitor and capture status for running jobs and tasks. Capturing the details allows us to determine how long a process takes, what the status of the process is, and if necessary, dive into Task level details. When a job runs, it generates a unique process (GGUID), which represents the running or ongoing work. The process will have a start and end time and will display the status of ongoing activities. Each task in the process will have a name, a start and end time, and a status. If a task fails, the process status will be marked as failed. If a user needs more visibility for a function, they can simply log the function with decorator and all details will be captured in dynamodb for that task. I will demonstrate how to design and implement these solutions.
- Shows all process that ran for given Day
- Shows all Task for Given Process for given Day (Query GSI View )
- Sends Alerts for failed process using kinesis streams lambda and SNS
Read Article : https://www.linkedin.com/pulse/smart-way-capture-monitor-report-status-python-jobs-using-soumil-shah
please email address in severless.yml Email address will recive alerts for failed process
MySubscription:
Type: AWS::SNS::Subscription
Properties:
Endpoint: your_email.com
Protocol: email
TopicArn: !Ref 'SNSTopic'
command 1: npm install -g serverless
command 2: serverless config credentials --provider aws --key XXXX --secret XXXXX -o
command 3: serverless deploy
class Jobs(object):
def __init__(self):
self.process_instance = Process()
self.__create_process()
def __create_process(self):
self.process_instance.create()
self.process_instance.progress()
def run(self):
response_1 = self.step_1()
response_2 = self.step_2()
self.process_instance.success()
@dynamodb_task()
def step_1(self):
print("some business rules and code goes here ")
print("some more business rules and code goes here ")
@dynamodb_task()
def step_2(self):
raise Exception ("OUCH")
print("some business rules and fucntion calls logs ")
-
@dynamodb_task() - Whenever you Decorate the method with dynamodb task this will log the meta data in dynamodb. this logs task start time and end time and status of task
- Status can be Success | Progress | Failed
-
GS1 gives you all Task for a given Process
-
GSI2 gives you all process for a guiven day and you can use SK to filter by status or any other things if needed
-
GSI3 gives you all process for month
-
TTL feilds will delete as the records and process get olders
Bachelor in Electronic Engineering | Masters in Electrical Engineering | Master in Computer Engineering |
-
Website : http://soumilshah.com/
-
Linkedin: https://www.linkedin.com/in/shah-soumil/
-
Youtube : https://www.youtube.com/channel/UC_eOodxvwS_H7x2uLQa-svw?view_as=subscriber
-
Facebook Page : https://www.facebook.com/soumilshah1995/
-
Email : [email protected]
-
projects : https://soumilshah.herokuapp.com/project
-
I earned a Bachelor of Science in Electronic Engineering and a double master’s in Electrical and Computer Engineering. I have extensive expertise in developing scalable and high-performance software applications in Python. I have a YouTube channel where I teach people about Data Science, Machine learning, Elastic search, and AWS. I work as data collection and processing Team Lead at Jobtarget where I spent most of my time developing Ingestion Framework and creating microservices and scalable architecture on AWS. I have worked with a massive amount of data which includes creating data lakes (1.2T) optimizing data lakes query by creating a partition and using the right file format and compression. I have also developed and worked on a streaming application for ingesting real-time streams data via kinesis and firehose to elastic search