Custom Software Engineer

Project Role : Custom Software Engineer Project Role Description : Develop custom software solutions to design, code, and enhance components across systems or applications. Use modern frameworks and agile practices to deliver scalable, high-performing solutions tailored to specific business needs. Must have skills : Grafana Good to have skills : Apache Kafka Minimum 7.5 year(s) of experience is required Educational Qualification : 15 years full time education Summary: As a Custom Software Engineer, a typical day involves developing tailored software solutions by designing, coding, and improving various components within systems or applications. The role requires working with modern frameworks and following agile methodologies to ensure the delivery of scalable and efficient software that meets unique business requirements. Collaboration with cross-functional teams and continuous refinement of software elements to enhance performance and functionality are integral parts of the daily workflow. Roles & Responsibilities: - Expected to be an SME, collaborate and manage the team to perform. - Responsible for team decisions. - Engage with multiple teams and contribute on key decisions. - Provide solutions to problems for their immediate team and across multiple teams. - Lead efforts to identify and implement process improvements that enhance team productivity and software quality. - Mentor junior team members to support their professional growth and integration within the team. - Coordinate with stakeholders to align software development activities with business goals and project timelines. Design and implement the observability layer for Akura across application, integration, event, agent, audit, and cloud runtime components. - Define logging, metrics, tracing, alerting, and dashboard standards for platform services. - Establish end-to-end correlation across Guidewire events, Kafka topics, backend services, agent decisions, ServiceNow actions, and APEX UI views. - Ensure all services expose health, readiness, latency, throughput, error, and dependency metrics. - Work with Solution Architect and DevOps Engineer to align observability with cloud deployment and operational readiness needs. - Configure and support OpenTelemetry instrumentation across backend microservices, Kafka consumers/producers, API services, agent services, and connectors. - Configure OpenTelemetry Collector pipelines for telemetry ingestion, enrichment, filtering, batching, memory limiting, masking, and routing. - Implement OTLP receivers and exporters as required by target observability platforms. - Use OpenTelemetry context propagation to preserve trace IDs and correlation IDs across services. - Support OpenTelemetry configuration governance and reusable collector patterns. - Instrument Spring Boot / FastAPI microservices with logs, traces, and metrics. - Capture API latency, request volume, error rate, dependency failures, and service health. - Define dashboard views for Guidewire Adapter Service, Incident Evaluation Service, ServiceNow Connector, Agent Orchestration Service, Audit Service, and APEX Backend APIs. - Ensure application logs include correlation ID, event ID, service name, environment, severity, and processing status. - Support troubleshooting of service-to-service and API gateway interactions. - Build dashboards and alerts for Kafka / Amazon MSK event processing. - Monitor topic throughput, consumer lag, failed events, retry counts, DLQ volume, partition health, and event processing latency. - Provide visibility into topics such as: guidewire.claim.event.created, incident.evaluation.requested, agent.incident.decision.created, incident.creation.requested, audit.event.recorded - Support replay and failure investigation by correlating Kafka offsets, event IDs, and audit entries. - Work with Backend/Kafka Engineers to define observability hooks for producers, consumers, and stream processing applications. - Implement observability for AI/agent workflows, including agent invocation count, agent latency, decision outcome, confidence distribution, failed tool calls, and human-in-loop routing. - Ensure agent decision flows are traceable from input event to final action. - Capture agent execution metadata, decision status, tool-call failures, and audit references. - Work with Agent Integration Engineer to expose metrics for incident triage, context enrichment, and recommendation workflows. - Build operational dashboards using CloudWatch, Grafana, or the client-approved observability tooling. - Create dashboard views for: Platform health, API health, Kafka processing, Agent decision flow, Incident creation status, Audit write success/failure, Deployment/runtime health, Business-unit-level event visibility - Work with platform and support teams to define alert thresholds and escalation views. - Support centralized observability and business-unit-specific dashboard patterns where required. - Define alerts for service failures, API latency, Kafka consumer lag, event processing failures, DLQ growth, audit failures, and agent action failures. - Support alert grouping, prioritization, and correlation to reduce operational noise. - Provide dashboards that help support users identify whether an issue is in Guidewire ingestion, Kafka processing, agent decisioning, ServiceNow integration, or APEX UI display. - Work with support and operations teams to define actionable alerts and runbook links. - Ensure telemetry does not expose sensitive data unnecessarily. - Implement masking, filtering, and routing controls for logs, metrics, and traces. - Support TLS/mTLS-based secure telemetry forwarding where applicable. - Work with Security/DevOps Engineer to align telemetry collection with security, privacy, and compliance requirements. - Support audit-friendly telemetry retention and access-control practices. - Define observability acceptance criteria for each platform component. - Create runbooks for alert investigation, serv