In this post I will focus on how to design the execution of Apex Asynchronous Processes (hereinafter Apex Async ), taking advantage of the capabilities of the platform.

But I will do it in an inverted way as usual, where we explain how to program the programming mechanisms. But in IMMO, if the developers, architects and administrators, we understand well how Salesforce manages the asynchronous processes, their Wait and Execution queues, their associated limits, etc., the choice of the programming mechanism is very simple or obvious.

In the asynchronous context, a design focused on the programming and not on the capabilities of the platform, may involve limiting the performance and performance of our project.

1. Introduction
2. Back to Basics: What is an asynchronous Apex process?
3. States of an asynchronous Job Apex
4. Apex Flex Queue (s) and the Apex Jobs Queue 
   4.1. Apex Flex Queue 
   4.1.1. Order of insertion in the Flex Queue and reordering of priorities 
   4.2. Test-Context Queue (the other Flex Queue) 
   4.3. Apex Jobs Queue
5. Queue limits
6. Monitoring of queues 
   6.1. Through user interface 
   6.2. Using the AsyncApexJob 
   6.3 object . Using CronTrigger and CronJobDetail for Jobs Scheduled
7. Real challenges not covered by the platform
8. Conclusions

2. Back to Basics: What is an asynchronous Apex process?

An Apex Job with asynchronous nature (hereinafter Apex Async or Job Async ), is executed when the platform has available resources (which happens very often) without any termination fork or ANS or ETC (Service Level Agreement – Estimated Time to Complete), and always without prejudice to the synchronous scope, where the end user that runs with priority works.

In contrast, the limits for a Job Async are upper limits than for a synchronous process. Thus an asynchronous Job possesses:

I stress, there is no ETC by Salesforce, and therefore, our design can not assume any of the following aspects:

But as you saw in the previous image, they are managed using queues. Therefore, here begins the Rock & Roll , let’s see the states of a process and how the platform manages the asynchronous Jobs through the queues.

1. When the process will start
2. How long the process will last
3. When the results of the execution will be available

There are different types of Jobs Apex Async, surely you have seen some method with the annotation @Future, but being very useful and practical, that is the younger brother of all.

There are 3 additional mechanisms, implemented through Interfaces (do not panic is very simple):

1. @Queueable
2. @Schedulable
3. @Batchable

3. States of an asynchronous Job Apex

The states of a process ( Job ) Apex Async, passes through the following states:

State that reports the platform	Description and Queue in which the Process resides	Is it a finalist state?
Holding company	The Batch process has been sent correctly, but it remains pending to be executed, when there are available resources, that allow its execution. The process resides in theFlex queue .	Do not
Queued	The execution is imminent. The process is transferred from the Flexqueue to the Jobs queue , and will remain in this queue until completion.	Do not
Preparing	The method startis executed, Rock & Roll has started.	Do not
Processing	The complete logic of the process is executed.	Do not
Aborted	It is a finalist state, which is reached if it is decided to abort the execution voluntarily.	Yes
Completed	The process has finished, there may be errors or completely correct completion.	Yes
Failed	State that indicates that something went wrong at the platform level, and the process was not completed correctly.	Yes

Let’s see how the execution of a Job relates to the tails of the platform.

4. The Apex Flex Queue (s) and the Apex Jobs Queue

The Salesforce platform offers 3 queues for the management of asynchronous processes:

1. Apex Flex Queue (Flex by Flexible)
2. Test-Context Queue (is the homologous of Flex for Testing)
3. Apex Jobs Queue (previously Apex Batch Queue)

Note: when Salesforce released the functionality of the Apex Jobs queue in 2015, it was talked about the Batch Queue, this term is currently deprecated and leads to confusion, so I will not use this terminology in this article, but the current one.

4.1. Apex Flex Queue

This queue is used to glue the Batch processes before inserting them into the Apex process execution queue. Allows for the pacification and reordering of these processes.

The procedure followed by the plaform is as follows:

1. Create the Job , serialize it to locate it in the system queues. If the creation was successful (limits, etc.), the system returns an ID identifier of the Job (this ID allows to inspect the characteristics of the Job)
2. Only processes of type Batch , are placed in the queue Flex Queue , with state Holding – this implies that it is not running – none of its methods has been invoked except the constructor of the class (this is important). As soon as the platform has available resources, it will execute the Job and move it from the Flex queue to the Apex Jobs queue.
3. Non-Batch asynchronous processes are inserted directly into the Apex Jobs queue , without going through the Flex queue

In the Flex queue, the process has all the attributes (15) of execution: Job Type, Summitted By, Apex Class, Apex Method , etc., that we can consult.

4.1.1. Order of insertion in the Flex Queue and rearrangement of priorities

The order of insertion in the queue is fundamental , since it will indicate how the system will initiate the execution of the Jobs. Therefore, the insertion order will mark the launch of the Job execution.

A common mistake is to consider that Batch processes will run one after the other. Since the platform can run up to 5 concurrent processes simultaneously, they can run in parallel taking advantage of available resources. Therefore, the designer of the solution should consider that:

1. The Jobs will be initiated through the FIFO (First In First Out) scheme.
2. Jobs can be executed simultaneously and the order of completion is unknown, since they are asynchronous processes using shared resources.

While the Job Batch is present in the Flex queue, without having started its execution, both the administrator through the user interface and the programmer through code, can alter the order of the processes, and in this way we alter the FIFO scheme.

The Administrator has a Queue Monitoring interface:

For its part, the Programmer has access to 4 methods of the class FlexQueue:

1. moveAfterJob(jobToMoveId, jobInQueueId)
2. moveBeforeJob(jobToMoveId, jobInQueueId)
3. moveJobToEnd(jobId)
4. moveJobToFront(jobId)

4.2. Test-Context Queue (the other Flex Queue)

The platform provides an additional queue, so that programmers can perform queuing tests on the Flex Queue. These methods belonging to the Test class are:

1. enqueueBatchJobs(numberOfJobs)that allows to send to the queue the indicated number of jobs
2. getFlexQueueOrder()which returns the list of the identifiers of the Jobs present in the queue

An example:

4.3. Apex Jobs Queue

It houses all the Jobs of the system, both in intermediate and non-finalist states (see previous section), both Salesforce internal Jobs and those created by code. These are:

Asynchronous Jobs :

1. Future
2. Queable
3. ScheduledApex
4. BatchApex

Jobs created by Salesforce for the internal management of the system:

1. SharingRecalculation
2. BatchApexWorker
3. TestRequest
4. TestWorker
5. ApexToken

Although not documented by Salesforce (as Pedro Espada comments in the Success Community), the processes registered with @Future are executed as a priority, ahead of Queueable and Batchable.

5. Limits of the queues

It is essential to know the limits on queues and Jobs. In my opinion the most important are:

1. There can be up to 100 Jobs in the Holding state . Therefore the maximum depth of the Flex queue, is 100 Jobs.
2. There can only be 5 Active Jobs , where Active means in the states: Queued, Preparing, Processing

1. Up to 50 Jobs can be sent to the Apex Job queue with the System.enqueueJob statement
2. Analogously for Jobs @Future we also have the limitation of 50 Jobs

Any insertion in queues that pretend to exceed these limits, receives an exception from the system, and is rejected as can be seen in the previous images.

6. How Tails Are Monitored

6.1. Through the User Interface

The Salesforce interface provides access to the Flex Queue and Apex Jobs processes.

In the case of the Flex queue, it also allows the reordering of the Jobs and access to their characteristics. The reordering is done by indicating in which position you want to place the Job.

6.2. Through queries on the AsyncApexJob object

In addition to the interface, the platform provides an object, AsyncApexJob , where each record describes a Job, with all its attributes. This record is updated as the Job goes through its life cycle.

This object accessible via SOQL that allows us to:

• get information about how many jobs we have
• how many are of each type
• how many are in a certain state
• get the duration of the finished ones
• see the characteristics of a concrete Job, etc.

The processes that reside in the Flex queue are also present in AsyncApexJob, since they are asynchronous Jobs like the rest, although in a certain state.

6.3. Using CronTrigger and CronJobDetail for Jobs Scheduled

For Jobs of Scheduled type , the platform also provides the CronTrigger and CronJobDetail objects (there is a lot of documentation in Trailhead and the API documentation of its use). 
Next I comment what I consider basic:

1. For CronTrigger the following fields must be known: PreviousFireTime , TimesTriggered , State , NextFireTime , etc., which contain the basic information of the executions of the Job regarding its executions
2. CronJobDetail contains 2 members: JobType and Name

7. Challenges not covered by the platform

Challenge 1: I have functionally dependent processes that must be executed in series

Knowing the life cycle of a Job and the queues used for its management, we can find a limitation very quickly: How to ensure that a Job is executed and previously completed to another, since there is a functional business dependence between them?

If we consider the initial Job as “Job father” and the dependent as “Job child”, we can translate the use case to the insertion of the Job Son in the Flex Queue should not be done until the execution of the Father process has concluded .

For this we have several alternatives:

1. If we have Jobs of type Queueable we can use Job Chaining
2. If we have Jobs of type Batchable , in the finish () of the father, we can launch the Job son, that is, apply the same scheme as Job Chaining but for Batchable
3. We synchronize the execution of the Jobs via Platform Events : when the Job Parent finishes, it launches an event so that the Job child is inserted in the Job Queue.

The 3 previous schemes, suffer from a defect, what happens if the child does not have only the dependence of a father but several dependencies, for example having several ancestors, and / or ancestors but not starting before 22.00 even if their ancestors were all finished at 20.00?

Therefore, as we expand our synchronization requirements, we see that the standard functionality limits us, for that reason, for me the solution is the programming of a dynamic scheduler , which we will see in this post .

Challenge 2: I have Jobs Scheduled that must be executed at a specific time or periodically in a guaranteed manner

This case is somewhat more complex. Its description to clarifying language is: ** When the moment of execution of the process, there must be an available slot, out of the 5 possible, for its execution, and thus have the maximum guarantees (not the total security) that this process it can be executed (remember that we can never know when an asynchronous process starts / ends and there are no associated SLAs or ETCs).

For this, we must have control of the processes that are active in the Apex Jobs queue and the processes in the Flex Queue.

This case of use, under my knowledge, again a dynamic planner that adequately controls both tails and their occupation can solve this challenge.

8. Conclusions

I do not know if I have succeeded, but in this post, I tried to provide a simple view of the types of asynchronous processes and their main differences with synchronous processes, how the platform manages them through queues, how to monitor their execution throughout their life cycle and I have finished analyzing certain scenarios that the platform does not directly solve.

In the following entries I will continue treating asynchronous processing:

• Select the most convenient asynchronous method in various real use cases
• A Dynamic Planner that allows to eliminate the limitations that the platform has, and that allows us to identify requirements that in our projects can be complex to satisfy

I hope you find it helpful, any comment is welcome.

Interesting links

1. Asynchronous Processing in Force com – Compulsory reading and comprehension
2. Asynchronous Processing Basics
3. Asynchronous Apex monitor
4. Monitoring the Apex Flex Queue
5. Monitoring the Apex Job Queue
6. SOAP API Developer Guide for AsyncApexJob
7. Differences between job types for ‘AsyncApexJob’ object
8. Trailhead – Async Apex
9. Trailhead – Control Processes with Queueable Apex