Chapter 2 : Backend Enterprise Ready Development

Enterprise Requirements for APIs and Microservices

Code Coverage - SonarQube

Generating Code Coverage Locally with JaCoCo, Sonar Scanner

Run Jacoco (Coverage Module)

Report Location

Coverage Report

Coverage Report : dbrest

Coverage Report : dbgraphql

Take-home Exercises
- Increase code coverage to 70-80%
  - Add more UT, IT
- Migrate Integration Tests from JUnit to TestNG

tests API Tests - TestNG, WebClient

dbrestTest - Tests Run

dbrestTest - Tests Run

dbrestTest - Report Location

dbrestTest - Report Location

dbrestTest - Report

dbrestTest - Report

dbrestTest - Emailable Report

dbrestTest - Emailable Report

dbgraphqlTest - Tests Run

dbgraphqlTest - Tests Run

dbgraphqlTest - Report

dbgraphqlTest - Report

dbgraphqlTest - Emailable Report

dbgraphqlTest - Emailable Report

Aggregate Test report

Aggregate Test report

API Logging

API Logging is included and available in emapi project.

e.g. API endpoint call - Query

curl -X 'GET' \
  'http://127.0.0.1:9080/emdbrest/tenant/Query?tenantId=3' \
  -H 'accept: application/hal+json'

Check API Logging

2024-10-02T16:57:21.469+05:30 DEBUG 35656 --- [io-9080-exec-10] o.s.w.f.CommonsRequestLoggingFilter      : Before request [GET /emdbrest/tenant/Query?tenantId=3]
2024-10-02T16:57:21.470+05:30 DEBUG 35656 --- [io-9080-exec-10] o.s.web.servlet.DispatcherServlet        : GET "/emdbrest/tenant/Query?tenantId=3", parameters={masked}
2024-10-02T16:57:21.471+05:30 DEBUG 35656 --- [io-9080-exec-10] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to com.example.emapi.app.Tenant.TenantDataRestController#TenantQuery(long)
2024-10-02T16:57:22.067+05:30 DEBUG 35656 --- [io-9080-exec-10] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Using 'application/hal+json', given [application/hal+json] and supported [application/json, application/*+json]
2024-10-02T16:57:22.067+05:30 DEBUG 35656 --- [io-9080-exec-10] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Writing [[Tenant [ tenantId = 3 ]]]
2024-10-02T16:57:22.071+05:30 DEBUG 35656 --- [io-9080-exec-10] o.s.web.servlet.DispatcherServlet        : Completed 200 OK
2024-10-02T16:57:22.072+05:30 DEBUG 35656 --- [io-9080-exec-10] o.s.w.f.CommonsRequestLoggingFilter      : API LOGGING: GET /emdbrest/tenant/Query?tenantId=3]

e.g. API endpoint call - Create with Payload

curl -X 'POST' \
  'http://127.0.0.1:9080/emdbrest/tenant/Create' \
  -H 'accept: application/hal+json' \
  -H 'Content-Type: application/json' \
  -d '  {
    "tenantId": 13,
    "customerId": 2,
    "tenantName": "Manipal - Pune Branch 2",
    "beginDate": "2024-09-19",
    "minOrderQuantity": 10,
    "customizationRequired": 1,
    "custMicrosvcType": "synchronous-webclient",
    "customizationService": "tenant-2-validation-service",
    "customizationPayload": "order_data"
  }'

Check API Logging with Payload

2024-10-02T17:06:11.229+05:30 DEBUG 35656 --- [nio-9080-exec-7] o.s.w.f.CommonsRequestLoggingFilter      : Before request [POST /emdbrest/tenant/Create]
2024-10-02T17:06:11.229+05:30 DEBUG 35656 --- [nio-9080-exec-7] o.s.web.servlet.DispatcherServlet        : POST "/emdbrest/tenant/Create", parameters={}
2024-10-02T17:06:11.229+05:30 DEBUG 35656 --- [nio-9080-exec-7] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to com.example.emapi.app.Tenant.TenantDataRestController#TenantCreate(Tenant)
2024-10-02T17:06:11.249+05:30 DEBUG 35656 --- [nio-9080-exec-7] m.m.a.RequestResponseBodyMethodProcessor : Read "application/json;charset=UTF-8" to [Tenant [ tenantId = 13 ]]
2024-10-02T17:06:11.488+05:30  INFO 35656 --- [nio-9080-exec-7] c.e.e.a.Tenant.TenantDataRestController  : Tenant Begin Create Record For [Tenant [ tenantId = 13 ]] 
2024-10-02T17:06:11.746+05:30  INFO 35656 --- [nio-9080-exec-7] c.e.emapi.app.Tenant.TenantService       : Tenant Created Record [Tenant [ tenantId = 13 ]] 
2024-10-02T17:06:12.229+05:30 DEBUG 35656 --- [nio-9080-exec-7] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Using 'application/hal+json', given [application/hal+json] and supported [application/json, application/*+json]
2024-10-02T17:06:12.229+05:30 DEBUG 35656 --- [nio-9080-exec-7] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Writing [Tenant [ tenantId = 13 ]]
2024-10-02T17:06:12.230+05:30 DEBUG 35656 --- [nio-9080-exec-7] o.s.web.servlet.DispatcherServlet        : Completed 201 CREATED
2024-10-02T17:06:12.231+05:30 DEBUG 35656 --- [nio-9080-exec-7] o.s.w.f.CommonsRequestLoggingFilter      : API LOGGING: POST /emdbrest/tenant/Create, payload=  {
    "tenantId": 13,
    "customerId": 2,
    "tenantName": "Manipal - Pune Branch 2",
    "beginDate": "2024-09-19",
    "minOrderQuantity": 10,
    "customizationRequired": 1,
    "custMicrosvcType": "synchronous-webclient",
    "customizationService": "tenant-2-validation-service",
    "customizationPayload": "order_data"
  }]

Note:

API Payload Logging is set for max length 10000. Adjust setting or disable it in file:
- emapi\lib\base-app\src\main\java\com\example\emapi\app\EmApiRequestLoggingFilterConfig.java

Security - OAuth2 with Keycloak

caution

Keycloak server needs to be accessible and running.
Run Keycloak in Docker
Using template for it, More info
- Tutorial 3 : EasyManage Templates

Run Keycloak in Docker

Run Keycloak in Docker

Build emapi project with Project Configuration,

Security Configuration :
- OAuth2 [Authorization Server]
Compile and Run project
Verify APIs security
Open Swagger API explorer and test API endpoints
- http://127.0.0.1:9080/swagger-ui/index.html
- Note: Swagger UI endpoints displaying is permitted without security.

Test APIs without auth token

curl -X 'GET' \
  'http://127.0.0.1:9080/emdbrest/erp_product/ViewAll' \
  -H 'accept: application/hal+json'

Server response:

Code    Details
401   Error: response status is 401

Log shows:

2024-10-02T17:50:42.428+05:30 DEBUG 36388 --- [nio-9080-exec-6] o.s.security.web.FilterChainProxy        : Securing GET /emdbrest/erp_product/ViewAll
2024-10-02T17:50:42.428+05:30 DEBUG 36388 --- [nio-9080-exec-6] o.s.s.w.a.AnonymousAuthenticationFilter  : Set SecurityContextHolder to anonymous SecurityContext
2024-10-02T17:50:42.428+05:30 DEBUG 36388 --- [nio-9080-exec-6] o.s.s.w.a.i.FilterSecurityInterceptor    : Failed to authorize filter invocation [GET /emdbrest/erp_product/ViewAll] with attributes [hasRole('ROLE_USER')]
2024-10-02T17:50:42.429+05:30 DEBUG 36388 --- [nio-9080-exec-6] o.s.s.w.s.HttpSessionRequestCache        : Saved request http://127.0.0.1:9080/emdbrest/erp_product/ViewAll?continue to session

Get auth token: Giving client_secret and password values

Note: Token expires in 60 sec by default

curl -d "client_id=login-app&client_secret=<client_secret>&username=emUser&password=<password>&grant_type=password" http://127.0.0.1:8180/realms/emapi/protocol/openid-connect/token

Representative Output: Trimmed for displaying here ...

{"access_token":"eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJNaU5KUmhtODZiYlZkUk82bGdiVC1EM3c5WEM4U2pkdnJBRndZYkVBaWFJIn0
...
rTmF-Ld1LMiksm8QBBsYGaZOKcWIiXXRuGhGD4RahbhnU6ckQoGD1LPnLNUHLwrTJT7YUW_1dCDMi95231q_MP4XFpmzCGpxBx4tRFmPf2Ag","expires_in":300,"refresh_expires_in":1800,"refresh_token":"eyJhbGciOiJIUzUxMiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIwN2UwZDk0O
...
kmU-2OxA","token_type":"Bearer","not-before-policy":0,"session_state":"81f96ffc-6a5f-484c-878e-d4456b3af966","scope":"email profile"}

Note auth token to use in curl below.

Test APIs with auth token

curl -H "Accept: application/json" -H "Authorization: Bearer <token>" "http://127.0.0.1:9080/emdbrest/erp_product/Query?productId=1"

Server response:

[{"productId":1,"productName":"string new","productCategory":"string new","primarySupplier":"string","productDesc":"string","productPicture":null}]

Log shows:

2024-10-02T18:11:30.987+05:30 DEBUG 36388 --- [nio-9080-exec-9] o.s.security.web.FilterChainProxy        : Securing GET /emdbrest/erp_product/Query?productId=1
2024-10-02T18:11:30.987+05:30 DEBUG 36388 --- [nio-9080-exec-9] o.s.s.o.s.r.a.JwtAuthenticationProvider  : Authenticated token
2024-10-02T18:11:30.987+05:30 DEBUG 36388 --- [nio-9080-exec-9] .s.r.w.a.BearerTokenAuthenticationFilter : Set SecurityContextHolder to JwtAuthenticationToken [Principal=org.springframework.security.oauth2.jwt.Jwt@82025aa3, Credentials=[PROTECTED], Authenticated=true, Details=WebAuthenticationDetails [RemoteIpAddress=127.0.0.1, SessionId=null], Granted Authorities=[ROLE_USER]]
2024-10-02T18:11:30.987+05:30 DEBUG 36388 --- [nio-9080-exec-9] o.s.s.w.a.i.FilterSecurityInterceptor    : Authorized filter invocation [GET /emdbrest/erp_product/Query?productId=1] with attributes [hasRole('ROLE_USER')]
2024-10-02T18:11:30.987+05:30 DEBUG 36388 --- [nio-9080-exec-9] o.s.security.web.FilterChainProxy        : Secured GET /emdbrest/erp_product/Query?productId=1
2024-10-02T18:11:30.987+05:30 DEBUG 36388 --- [nio-9080-exec-9] o.s.w.f.CommonsRequestLoggingFilter      : Before request [GET /emdbrest/erp_product/Query?productId=1]
2024-10-02T18:11:30.987+05:30 DEBUG 36388 --- [nio-9080-exec-9] o.s.web.servlet.DispatcherServlet        : GET "/emdbrest/erp_product/Query?productId=1", parameters={masked}
2024-10-02T18:11:30.987+05:30 DEBUG 36388 --- [nio-9080-exec-9] s.w.s.m.m.a.RequestMappingHandlerMapping : Mapped to com.example.emapi.app.ErpProduct.ErpProductDataRestController#ErpProductQuery(long)
2024-10-02T18:11:31.507+05:30 DEBUG 36388 --- [nio-9080-exec-9] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Using 'application/json', given [application/json] and supported [application/json, application/*+json]
2024-10-02T18:11:31.507+05:30 DEBUG 36388 --- [nio-9080-exec-9] o.s.w.s.m.m.a.HttpEntityMethodProcessor  : Writing [[ErpProduct [ productId = 1 ]]]
2024-10-02T18:11:31.507+05:30 DEBUG 36388 --- [nio-9080-exec-9] o.s.web.servlet.DispatcherServlet        : Completed 200 OK
2024-10-02T18:11:31.507+05:30 DEBUG 36388 --- [nio-9080-exec-9] o.s.w.f.CommonsRequestLoggingFilter      : API LOGGING: GET /emdbrest/erp_product/Query?productId=1]

If token expired, The logs show

2024-10-02T18:08:11.335+05:30 DEBUG 36388 --- [nio-9080-exec-3] o.s.s.oauth2.jwt.JwtTimestampValidator   : Jwt expired at 2024-10-02T12:34:29Z
2024-10-02T18:08:11.335+05:30 DEBUG 36388 --- [nio-9080-exec-3] o.s.s.o.s.r.a.JwtAuthenticationProvider  : Failed to authenticate since the JWT was invalid

Security - OAuth2 with Keycloak With OAuth2 Client

Take-home Exercises
- Implement Security Requirements for below.
- OAuth2 [Authorization Server + OAuth2 Client For Web Apps]
- OAuth2 [Authorization Server + OAuth2 Client For BFF Auth APIs]

Redis cache

caution

Redis server needs to be accessible and running.
Run Redis in Docker
Using template for it, More info
- Tutorial 3 : EasyManage Templates

Run Redis in Docker

Run Redis in Docker

Build emapi project with Project Configuration,

Redis Cache : Include [✅] Enable [✅]

Enable Swagger UI to display Request Duration Swagger UI has the display Request Duration parameter to show how long a request takes. Enable it.
- Edit : emapi\app\dbrest\src\main\resources\application.properties
- Add line springdoc.swagger-ui.display-request-duration=true
Compile and Run project
Verify APIs response times
Open Swagger API explorer and test API endpoints
- http://127.0.0.1:9080/swagger-ui/index.html

Test APIs : 1st call

http://127.0.0.1:9080/emdbrest/erp_customer/Query?customerId=5

Server response in:
Request Duration 
1045 ms

Test APIs : 1st call

Test APIs : 1st call

Test APIs : 2nd call - Which will use redis cached data!

http://127.0.0.1:9080/emdbrest/erp_customer/Query?customerId=5

Server response in:
Request Duration 
104 ms

Test APIs : 2nd call

Test APIs : 2nd call

Serverless

Covered in tutorial : Serverless

Logging - ELK stack

caution

ELK Stack consists of Elasticsearch, Logstash, and Kibana
ELK servers needs to be accessible and running.
Run ELK in Docker
Using template for it, More info
- Tutorial 3 : EasyManage Templates

Run ELK in Docker

Run ELK in Docker

Build emapi project with Project Configuration,

Logging - ELK : Include [✅] Enable [✅]

Add custom logs for observability and monitoring, e.g.:

    logger.info(String.format("ErpCustomer list size=[%s] firstCustomerId=[%s] firstCustomerName=[%s]", ErpCustomerList.size(), ErpCustomerList.get(0).getCustomerId(), ErpCustomerList.get(0).getName()));

Compile and Run project
Verify APIs
Open Swagger API explorer and test API endpoints
- http://127.0.0.1:9080/swagger-ui/index.html

Log shows:

logdate=(2024-10-02T19:25:05,046) thread=(main)) level=(DEBUG) loggerclass=(org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver) message=(ControllerAdvice beans: 2 @ExceptionHandler, 1 ResponseBodyAdvice)
logdate=(2024-10-02T19:25:05,091) thread=(main)) level=(DEBUG) loggerclass=(org.springframework.web.servlet.mvc.method.annotation.ExceptionHandlerExceptionResolver) message=(ControllerAdvice beans: 2 @ExceptionHandler, 0 ResponseBodyAdvice)
logdate=(2024-10-02T19:25:05,593) thread=(main)) level=(INFO) loggerclass=(com.example.emapi.app.EmDbRestAppRestSpringApp) message=(Started EmDbRestAppRestSpringApp in 9.93 seconds (process running for 11.36))

Test APIs : Make Create calls, with "productId" = 800, 801, 802

curl -X 'POST' \
  'http://127.0.0.1:9080/emdbrest/erp_product/Create' \
  -H 'accept: application/hal+json' \
  -H 'Content-Type: application/json' \
  -d '{
  "productId": 802,
  "productName": "Product 802",
  "productCategory": "Vitamin",
  "primarySupplier": "Cipla"
}'

View Logs in Kibana

Access from Kibana UI: http://localhost:5601

Configure index pattern: logstash-*
Visualize : Create Visualization
- View

DevOps - CI/CD

DevOps CI/CD pipelines for build, verify, deploy - using Jenkins.
Covered in tutorial : DevOps - CI/CD

Chapter 2 : Backend Enterprise Ready Development

Enterprise Requirements for APIs and Microservices​

Code Coverage - SonarQube​

tests API Tests - TestNG, WebClient​

API Logging​

Security - OAuth2 with Keycloak​

Security - OAuth2 with Keycloak With OAuth2 Client​

Redis cache​

Serverless​

Logging - ELK stack​

View Logs in Kibana​

DevOps - CI/CD​

Enterprise Requirements for APIs and Microservices

Code Coverage - SonarQube

tests API Tests - TestNG, WebClient

API Logging

Security - OAuth2 with Keycloak

Security - OAuth2 with Keycloak With OAuth2 Client

Redis cache

Serverless

Logging - ELK stack

View Logs in Kibana

DevOps - CI/CD