Containers Trend Report. Explore the current state of containers, containerization strategies, and modernizing architecture.
Securing Your Software Supply Chain with JFrog and Azure. Leave with a roadmap for keeping your company and customers safe.
Software design and architecture focus on the development decisions made to improve a system's overall structure and behavior in order to achieve essential qualities such as modifiability, availability, and security. The Zones in this category are available to help developers stay up to date on the latest software design and architecture trends and techniques.
Cloud architecture refers to how technologies and components are built in a cloud environment. A cloud environment comprises a network of servers that are located in various places globally, and each serves a specific purpose. With the growth of cloud computing and cloud-native development, modern development practices are constantly changing to adapt to this rapid evolution. This Zone offers the latest information on cloud architecture, covering topics such as builds and deployments to cloud-native environments, Kubernetes practices, cloud databases, hybrid and multi-cloud environments, cloud computing, and more!
Containers allow applications to run quicker across many different development environments, and a single container encapsulates everything needed to run an application. Container technologies have exploded in popularity in recent years, leading to diverse use cases as well as new and unexpected challenges. This Zone offers insights into how teams can solve these challenges through its coverage of container performance, Kubernetes, testing, container orchestration, microservices usage to build and deploy containers, and more.
Integration refers to the process of combining software parts (or subsystems) into one system. An integration framework is a lightweight utility that provides libraries and standardized methods to coordinate messaging among different technologies. As software connects the world in increasingly more complex ways, integration makes it all possible facilitating app-to-app communication. Learn more about this necessity for modern software development by keeping a pulse on the industry topics such as integrated development environments, API best practices, service-oriented architecture, enterprise service buses, communication architectures, integration testing, and more.
A microservices architecture is a development method for designing applications as modular services that seamlessly adapt to a highly scalable and dynamic environment. Microservices help solve complex issues such as speed and scalability, while also supporting continuous testing and delivery. This Zone will take you through breaking down the monolith step by step and designing a microservices architecture from scratch. Stay up to date on the industry's changes with topics such as container deployment, architectural design patterns, event-driven architecture, service meshes, and more.
Performance refers to how well an application conducts itself compared to an expected level of service. Today's environments are increasingly complex and typically involve loosely coupled architectures, making it difficult to pinpoint bottlenecks in your system. Whatever your performance troubles, this Zone has you covered with everything from root cause analysis, application monitoring, and log management to anomaly detection, observability, and performance testing.
The topic of security covers many different facets within the SDLC. From focusing on secure application design to designing systems to protect computers, data, and networks against potential attacks, it is clear that security should be top of mind for all developers. This Zone provides the latest information on application vulnerabilities, how to incorporate security earlier in your SDLC practices, data governance, and more.
Microservices and Containerization
According to our 2022 Microservices survey, 93% of our developer respondents work for an organization that runs microservices. This number is up from 74% when we asked this question in our 2021 Containers survey. With most organizations running microservices and leveraging containers, we no longer have to discuss the need to adopt these practices, but rather how to scale them to benefit organizations and development teams. So where do adoption and scaling practices of microservices and containers go from here? In DZone's 2022 Trend Report, Microservices and Containerization, our research and expert contributors dive into various cloud architecture practices, microservices orchestration techniques, security, and advice on design principles. The goal of this Trend Report is to explore the current state of microservices and containerized environments to help developers face the challenges of complex architectural patterns.
Observability Maturity Model
This is an article from DZone's 2023 Containers Trend Report.For more: Read the Report There are few technologies that have seen as much attention and adoption over the last decade as Infrastructure as Code (IaC) and containers. In those years, both have improved and eventually converged to form a modern standard for application delivery. In this article, we'll take a look at this transformation, the ways it has uprooted traditional methods of working in software, and how DevOps top performers use the technologies to deliver value at a blistering pace. Two Powerful Technologies IaC and containers emerged initially to solve different sets of problems. Modern IaC was predated by configuration management, aiming to gain some control over the frequently unpredictable drift of infrastructure over time. Not only did automation increase the ease with which infrastructure could be provisioned, it improved stability as well because services could be quickly recreated instead of manually fixed. Yet, in a world where dev and ops are split, advances in infrastructure automation remained squarely in the toolkit of ops. Containers were also born of a need for reproducibility, though in the context of applications. As installed libraries, dependencies, and other aspects of the runtime environment changed over time or across machines, the behavior of applications hosted on them changed as well. Many early approaches leveraged the Unix chroot syscall, which eventually converged to the development of containers as a more complete abstraction. Unlike IaC, much of the pain that containers solved was shared by both dev and ops. Collaboration across different workstations was challenging for dev and deploying that application to yet another machine in production was difficult for ops. Breaking Down the Wall of Confusion A silver lining of previously separate groups having the same problem is they start to work more closely to find a solution. The convergence within the industry toward containers has tracked alongside the DevOps movement, breaking down many of the traditional silos found in companies. Now an icon of this transformation, the "wall of confusion" connotes the challenges faced in traditional organizations when work moves in one direction from dev to ops. In the most pathological scenario, code that "works" on a developer's machine is "tossed over the wall" to be deployed and maintained by the operations team. Figure 1 illustrates this scenario: Figure 1: The wall of confusion Work moves from dev to ops: dev is responsible for Green, and ops is responsible for Red Success in this situation is difficult to achieve for many reasons, including that: It is not clear who should compile the binary; the developer understands application requirements but operations knows the real target environment. Many details of the environment that the application was tested in are known, but only to the developer. Operations must recreate this environment without having a consistent way to document it precisely. Even with IaC on the ops side, untracked changes to the dev environment will lead to production failures. Without containers, the use of IaC can improve outcomes only marginally. Ops can manage and change all that is on their side of the wall more effectively, but the same gaps in communication and feedback are still present. The core limitation is not lifted, and this is demonstrated well by the scope of work needed to be done by IaC tools. Many steps to prepare dependencies for the application are redone by ops when provisioning the production environment, even though the developer already took similar steps to produce their development environment. IaC tools will build the environment quickly, but without effective communication channels, they will only build broken environments more quickly. Once containers are introduced, however, these limitations can be significantly diminished. Figure 2: Delivering from dev to ops with containers Containers enable a more direct line of flow between dev and ops After containerizing the application, many previously hidden details can be shared explicitly: Most of the application dependencies are delivered directly to the production environment. The binary can be compiled, linked, and tested before handoff. The configuration can be split into what is already known at build time and what must be determined when the application is deployed. The overall footprint of what ops must independently build and maintain is significantly reduced. The minimization of hidden dependencies on the developer's machine has an outsized impact on the shared understanding across the organization. Of the three remaining dependencies depicted in this scenario, source code is largely irrelevant once the downstream artifacts have been persisted in the container. The runtime configuration and hardware dependencies remain as the key dependencies outside the containerized application. The definition of these is a natural task for many IaC tools, specifying resources such as CPU and networking supplemented with relevant connection parameters and credentials. It is also more appropriately aligned with the concerns of operations. Doubling Down on Containers The trend of container adoption in the industry has had an impact on the way dev and ops are able to work, and has been central to many DevOps transformations. What's next for these technologies? In the same way these technical advances have changed the way we work, the changes to the way we work have paved the way for further improvements to the technology. The abstraction that containers provided ultimately reduced the set of concerns that IaC tools had to deal with. This enabled the development of better abstractions for IaC, largely built around virtualized compute, storage, and networking. Installing dependencies and managing OSs has shifted left into the container build process and often within the development cycle. In many ways, containers form a contract between the application and the infrastructure that will run it. This has been formally defined by the Open Container Initiative (OCI), and an ecosystem of modern cloud-native tools has emerged conforming to the specification. A familiar but often overlooked element of container-driven architectures is the container registry. By systematically submitting container images to a registry, the DevOps workstream gains the additional benefit of decoupling software delivery. Figure 3: Decoupling build and deploy Dev may build container images with Docker, and ops may deploy them with Kubernetes A container image, representing an immutable snapshot of a deployable application, is pushed to the registry. The application can then be deployed to any OCI-compliant container runtime. This may be directly to a virtual machine (VM), an open-source orchestration framework like Kubernetes, or one of the many managed container runtimes offered by cloud providers. A common element between these options is that container images are pulled from the registry when needed. Not only does this enable greater control over when changes are actually released, but it also provides a natural means of declaratively specifying the application being deployed. In fact, this is arguably the key to achieving some of the longest-held goals of IaC. The entire infrastructure is declared authoritatively with immutable references to applications. This is the closest the industry has been able to get to a single definition of the entire system. Modern Patterns Made Possible As defined by the OpenGitOps project, the core principles of GitOps are: Declarative Versioned and immutable Pulled automatically Continuously reconciled With these principles in mind, it is clear that GitOps is a natural extension of patterns enabled by IaC and containers. The system definition is made declarative and immutable through the use of IaC with references to immutable containers. By persisting the definition to a version control system such as Git, we can achieve the versioned quality. The remaining two principles — pulled automatically and continuously reconciled — are the extensions that GitOps offers and can generally be implemented by running agents in the target environment. One agent is responsible for synchronizing the latest state from source control and the other ensures that the actual state continues to match the desired state. In order for an organization to fully benefit from a microservices architecture, the services must be small and independently deployable. Deployment pipelines have often been built to push builds through various stages, sometimes making it difficult to selectively release only certain parts of the system. As we have seen, container registries offer essential decoupling: Build pipelines push images to the registry, and the declared infrastructure pulls those images precisely when we want it to. This enables isolation between teams and small, safe changes to production. Conclusion Starting initially as tools meant for distinct use cases, IaC and containers have converged to form a modern framework for cloud-native application delivery. With maturing open-source standards, they are democratizing access to previously elite levels of quality and reliability. First-class support from cloud providers is further reducing barriers to adoption. The greatest advances are in the ways we work and collaborate as teams, and this is ultimately how we are able to deliver the greatest value and at the highest levels of quality. This is an article from DZone's 2023 Containers Trend Report.For more: Read the Report
This is an article from DZone's 2023 Containers Trend Report.For more: Read the Report Edge computing and containers have become increasingly popular in recent times, providing innovative solutions to various challenges related to data processing in our daily lives. These technologies have now permeated a wide range of devices, including our cars, phones, and even refrigerators, unlocking new possibilities for use cases and enabling us to address data processing challenges more efficiently. In this article, we will explore the intersection of edge computing and containers, their importance, and the challenges associated with them. Use Cases for Edge Computing and Containers There are several industries that can benefit from the use of edge computing and containers, including the Industrial Internet of Things (IIoT), healthcare, smart cities, and retail. Figure 1: Edge computing and containers use cases Edge computing is everywhere, and rarely are industries not using it. Here are the more mature ones: Application Domain Edge Computing Use Case IIoT in manufacturing Collect and process real-time data from sensors. Deploy lightweight applications for monitoring machine performance and triggering alerts on the edge devices. Can also be used for malfunction detection, increasing performance, performing quality checks on a newly created product, etc. Healthcare For example, in a hospital, wearable devices can be used to collect data on patient vitals, which can be processed in real time using edge computing. In other cases, programs integrated into x-ray scanners can diagnose cancers and other illnesses (better than humans). Smart cities Sensors and cameras can be used to collect data on traffic flow, air quality, and other factors. Containers can be used to deploy and manage applications that analyze this data and provide insights to city planners. In some countries, they try to detect illegal behavior, too. Retail In a retail store, sensors and cameras can be used to collect data on customer behavior, such as which products they are looking at or how long they spend in certain areas of the store. We also have fully automated shops without cashiers. The Intersection of Edge Computing and Containers Edge computing and containers have several commonalities, including their ability to support distributed applications and their focus on reducing latency. Containers are particularly well suited for edge computing because they are lightweight and can be easily deployed in remote locations. However, there are also challenges associated with using containers in edge computing environments, such as limited resources and security concerns. Benefits of edge containers include: Flexibility – Edge containers are highly portable and can run on various edge devices, providing flexibility and agility in deployment. Scalability – Containers are highly scalable and can be quickly replicated, deployed, and managed across multiple edge devices, making it easier to scale applications and services. This is particularly important in edge computing environments where resources are limited and traditional monolithic applications may not be practical. Security – Containers provide a secure environment for running applications and isolating them from other processes on the edge device. Low latency – By processing data closer to the source, containers can help reduce the amount of time it takes for data to travel between devices and data centers. This is particularly important in applications that require real-time processing, such as those used in IIoT or healthcare. Reduced bandwidth – Centralized applications often result in high network charges due to the concentration of all traffic within the cloud vendor's data center. On the other hand, edge containers can be located closer to the end user, allowing for pre-processing and caching of data, which can help reduce network charges. Maturity – Docker, as a container technology, is considered to be stable and widely used in production environments. Moreover, developers can leverage their existing knowledge and skills using Docker, which means no additional training is required when testing edge containers. Challenges of edge containers include: Limited resources – Edge devices often have limited resources such as memory, processing power, and storage, which can impact the performance of edge containers. Complexity – Edge containers require expertise in containerization and distributed computing, which can be challenging for some organizations. Management – Managing containers across multiple edge devices can be complex and time consuming, requiring robust container orchestration solutions. Security – Edge devices are often located in remote and unsecured locations, which can make them vulnerable to attacks. Containers can also introduce security risks, such as container breakouts or vulnerabilities in container images. Implementing Edge Computing and Containers Edge computing and containers offer numerous benefits that organizations quickly adopt for their different business cases. However, successfully implementing these technologies requires careful consideration of several key factors. Choosing the Right Container Platform When implementing edge computing and containers, it is important to choose the right container platform (e.g., Docker). These platforms provide a range of features and capabilities, such as container orchestration and management, which can help simplify the deployment and management of containers in edge computing environments. Still, the common widely used platforms, such as Kubernetes and OpenShift, are not adapted to edge computing due to the resources capacity of the edge devices. It’s recommended to switch to compatible alternatives, often open source, such as k3s, KubeEdge, microk8s, or Baetyl. Deployment Strategies Deployment strategies should be considered when implementing edge computing and containers. Depending on the specific use case, organizations may choose to use a hybrid cloud model, where some services are deployed in the cloud and others are deployed on edge devices. Alternatively, containers can be deployed directly on edge devices, which can help reduce latency and improve performance. Once deployed, managing edge computing and containers can be challenging, particularly in environments with a large number of edge devices. Container orchestration and management platforms (e.g., Kubernetes) can help simplify the management of containers in edge computing environments. These platforms provide features like automatic scaling, load balancing, and health monitoring, which can help ensure that containers are running efficiently and effectively. In addition, monitoring container performance/status is critical for identifying and addressing issues before they become major problems. This includes monitoring container resource usage, network traffic, and application performance, and using tools like logs and metrics to troubleshoot issues. Open-source tooling such as OpenTelemetry and Prometheus are often a good starter pack. Plan for edge device failures: Edge devices can fail unexpectedly, so it's important to plan for such scenarios by implementing redundancy measures, such as running multiple instances of containers across different edge devices or using edge-to-cloud failover mechanisms. Security Considerations Security considerations are important when implementing edge computing and containers. Edge computing relies on numerous devices and networks that are vulnerable to cyber-attacks, including malware, ransomware, and phishing attacks. Without proper security measures, these devices and networks can be compromised, leading to data breaches and other security incidents. If an edge device is compromised, it could infect the whole network. Another challenge is data protection, especially when sensitive data is involved and you can hardly prevent physical access to the device. Finally, the lack of standardization in edge computing can create security challenges by making it more difficult to implement consistent security measures across devices and networks. Security remains the main challenge when using edge computing, and it can require a lot of effort to mitigate risks. The Future of Edge Computing and Containers The future of edge computing and containers is promising, with emerging trends like the use of artificial intelligence (AI) and machine learning, as well as the development of new container technologies. For example, edge devices are increasingly being equipped with AI and machine learning capabilities, which can help improve the accuracy and speed of data processing. Today, we can already get self-driving cars, smart cameras that can tell the difference between a cat/dog or a person (thief), automated sorting machines in the recycling industry, or simply your watch, which can analyze your health data and detect a heart attack. All of these leverage edge computing coupled with AI, and the number of use cases in our daily life will increase quickly in the upcoming years. In parallel, to keep pace with these new use cases, new container technologies such as WebAssembly are also being developed, which can help improve the performance and security of containers in edge computing environments. A study of the edge computing market announced a 20-30% year-over-year (YoY) growth for the next 10 years, confirming the potential of the technology. Big tech companies will invest in the implementation brought by simplified deployment solutions that will empower every industry to use them. Figure 2: U.S. edge computing market (Data source) The impact of edge computing and containers on businesses and society will continue to grow, with new opportunities for innovation and efficiency. For example, in agriculture, we could see devices placed in the fields to collect data such as soil moisture, temperature, and humidity. This data can then be processed in real time using AI algorithms to optimize irrigation, fertilizer use, and pest management, leading to higher crop yields and reduced environmental impact. All types of equipment could use relevant sensors to optimize energy usage, while others could be equipped with AI algorithms that analyze data from sensors and other sources to detect potential equipment failures before they occur. This can help reduce downtime and maintenance costs, as well as improve overall equipment performance. Conclusion While there are challenges associated with the intersection of edge computing and containers, the opportunities for innovation and efficiency are significant. As more industries adopt these technologies, it is important to consider the challenges and opportunities associated with their implementation. By choosing the right container platform and deployment strategy, as well as making the best security decision, organizations can successfully implement edge computing and containers to drive business value. The future of edge computing and containers is promising, with new technologies and use cases emerging all the time. By staying up to date with these trends, organizations can continue to innovate and drive value in their respective industries. This is an article from DZone's 2023 Containers Trend Report.For more: Read the Report
Sometimes, it is necessary to examine the behavior of a system to determine which process has utilized its resources, such as memory or CPU time. These resources are often scarce and may not be easily replenished, making it important for the system to record its status in a file. By doing so, it becomes feasible to identify the most resource-intensive process in the past. If the system has not encountered an Out-of-Memory (OOM) killer, which can be found in the syslog, this information can be used to further pinpoint the problematic process. Atop Tool: An Overview There is a special tool that can be used both for real-time monitoring system usage and collecting system status into logs in the background. This is atop. With atop, you can gather information on CPU and memory usage, which can also be collected by other popular monitoring tools like top and htop. Additionally, atop provides insights into I/O and network usage, eliminating the need to install additional tools for network and I/O monitoring, such as iftop and iostat. In my opinion, atop is a versatile tool for many tasks. Atop is an open-source project and is available for most Linux distributions. What Is Atop Used For? Atop can be used for incident investigations in a Linux environment. Atop is a system resource monitor that can provide detailed information about system activity, including CPU, memory, and disk usage, as well as process-level activity During an incident investigation, atop can help you identify which processes were running at the time of the incident, how many resources they were consuming, and whether there were any spikes in resource usage that may have contributed to the incident. You can also use atop to monitor specific system components, such as network activity, and track changes over time. Basic use cases are listed below: Real-time resources monitoring Incidents analysis of the system behavior Capacity planning Resource allocation For most of the cases in the list, you can use modern monitoring systems like Zabbix and Prometheus. In my personal experience, I find atop to be a useful tool for troubleshooting and identifying the root cause of issues. While special monitoring systems can provide consolidated data on resource usage, they may not be able to answer specific questions about which processes led to server inaccessibility. Atop, on the other hand, can provide detailed information on individual processes, making it easier to differentiate between them and understand their impact on system performance. General principles working with atop: Real-time monitoring Incident investigation The first approach can be helpful for debugging or profiling your application, providing insights into its behavior and performance. On the other hand, the second approach is more useful for incident investigations, allowing you to identify the root cause of system failures or performance issues. Setting Up For writing logs, you should launch a demon: Shell # systemctl start atop It is recommended to change the interval for collecting data: Shell # vi /lib/systemd/system/atop.service You can find the env variable: Shell LOGINTERVAL=60 Change this value (in seconds) and reload the systemd unit configuration: Shell # daemon-reload Then start: Shell # systemctl start atop After that, atop will write info into a log file every 60 seconds (as above). Real-Time Monitoring Practical Examples Launching 1. To launch the utility type: Shell # atop In a terminal and track resource consumption: 2. In order to change the interval, press 'I' and enter the number in seconds: I prefer to set up an interval of 1-2 seconds. 3. In case the consumption of server resources reaches a critical value, it will be marked with a specific color: Red if consumption is critical Cyan if consumption is almost critical(80% of critical) The amount considered critical varies for different resources: 90% utilization of CPU 70% usage of disk 90% of network bandwidth 90% of memory occupation 80% of SWAP Of course, these parameters can be modified. Pay attention, the CPU has two cores, and you can see utilization distribution among these cores. 4. For killing a process, press ‘k’ and then type a PID of the process to be killed(it’s similar to ‘top’). Further, you can specify a signal to be sent to a process. Output Options Resource Related Output 1. To show commands how they have been run, type ‘c’: 2. If you would like to show all about memory, use the ‘m’ key: 3. There is ‘g’ for showing generic output. It might be needed when you want to revert to initial output. This is the default output. 4. For output of disk things, press ‘d’: 5. Network-related output (UDP, TCP, and bandwidth). For this, press ‘n’: Please, take into account that a kernel module netatop must be installed. Otherwise, atop won’t be out network-related information. This module allows us to show network activity per process. Refer to the official web page. So, we considered basic options, which is enough for most cases. Also, there are interesting options I recommend considering: ‘y’ — for showing per thread. It is a very useful functionality for examining the behavior of multi-threaded applications(or for debugging such apps). ‘e’ — shows GPU utilization ‘o’ — if you’d like to customize the output, it’s possible in ~/.atoprc, then you can use your own output just by pressing ‘o’ ‘z’ — if you need to pause your atop Aggregation Functions Top of Resources Eaters 1. Switch to show output accumulated per user, push ‘u’: 2. Output per process, hit ‘p’: 3. For output processes accumulated per Docker container, there is ‘j’ key: Where ‘host’ — host native processes. For observing only a specific container, use ‘J’ for this. Sorting Options 1. For sorting by CPU usage, press shift + ‘c’(or capital C) This is default behavior. 2. Sort by memory usage, hit shift + ‘m’(capital M) 3. Sort by disk usage, hit shift + ‘d’(capital D) 4. Network utilization sorting, use shift + ‘n’ (capital N) 5. If you are tracking threads, there is option ‘Y’ to aggregate threads by the process. Note. Sorting and output modifiers are different and should be used in combination. Incidents Examining (Looking to the Past) All those rules for real-time monitoring work for looking for events in logs. Initially, we need to start reading logs instead of real-time status output: Shell # atop -r /var/log/atop/atop.log Will read the log file. Navigating Navigate within the file using the t (forward) and shift+t keys (back). This allows you to go to the next sample or go back to the previous one. Time Limit There are options to limit time: Shell # atop -r /var/log/atop/atop.log -b 1400 Opens atop from 14:00 of the current day to the end of the current log file: <screencast> Shell # atop -r /var/log/atop/atop_20230523.log -b 1400 Opens file written on 25 of May 2023 year after 14:00, and navigates until 23:59 of the 25 of May: <screencast> Shell # atop -r /var/log/atop/atop_20230525 -b 14:00 -e 16:00 You’ll see records from 14:00 until 16:00 written on 25 of May 2023: <screencast> In case your system does not rotate logs, you can use atop's begin and end limitations in such view: Shell [-b [YYYYMMDD]hhmm ] [-e [YYYYMMDD]hhmm ] As was told above, sorting, aggregating data, and showing specific output related to some resources all these work perfectly in this mode. Other Atop Capabilities Atop has a unique feature that allows users to create charts directly in their terminal. To use this feature, you need only Python and pip, then install a specific package atopsar-plot, and you are able to visualize historical data. While this feature may not be particularly useful for modern systems that are already under monitoring, it's worth noting as an additional capability of the program. Monitor a Process Resource Consumption When it comes to monitoring a server, having the right tools in place is crucial to ensure optimal performance and identify potential issues. Two popular systems for server monitoring are Zabbix and Prometheus, both of which are capable of monitoring various process resources consumptions such as memory, CPU, and disk usage. These systems can extract information about a process from the /proc filesystem and send it to the server for storage. I should tell you monitoring systems extract info about spending resources by a specific process only or totally by all processes with no differentiation. Atop, in this case, is a powerful tool. Atop vs. Top While both atop and top are system performance monitoring tools, they differ in their capabilities and level of detail. Top is a simple command-line utility that provides a basic overview of the system's current processes and their resource usage. It is useful for quickly identifying processes that are consuming significant resources, but it does not provide detailed information on system activity. Atop, on the other hand, provides a more detailed report of system activity, including CPU usage, memory usage, and disk I/O. It can also monitor system activity over a period of time, making it useful for analyzing long-term trends and identifying patterns. Conclusion Atop is a powerful tool for system performance monitoring and analysis. It provides detailed information on system activity and can be used to diagnose and troubleshoot performance issues, plan for future capacity requirements, monitor security and compliance and allocate resources effectively. While it may be more complex than traditional tools like top, it offers greater insight into system activity and can be an invaluable tool for system administrators and IT professionals.
In this tutorial, we will learn how to deploy and interact with an Ethereum Smart contract using Web3j and Spring. What Is a Smart Contract? A Smart contract is an algorithm for certain actions integrated into the blockchain code. It is deployed on the Ethereum blockchain network and automatically executes predefined actions when specific conditions in the contract are met. What Is Solidity? Solidity is an object-oriented programming language for creating Smart contracts. Learning and using Solidity is easy if you are already familiar with Java or C. Solidity has built-in features specifically related to the blockchain. They allow you to withdraw and send "money" (ETH), get the address of the person who invoked the Smart contract, and make calls to other Smart contracts using their addresses. What Is Web3j? Web3j is a lightweight Java library that allows you to work with the Ethereum blockchain, providing the ability to manage transactions and generate type-safe wrappers for Smart contracts. 1. Installing Solc and Web3j Solc is the compiler for Solidity. To install it, run the following command: Shell npm install -g solc To generate the wrapper code of a Smart contract, we need to install Web3j: Shell curl -L get.web3j.io | sh && source ~/.web3j/source.sh 2. Initializing the Spring Project To quickly start a project, you can use Spring Initializr: Download the project by clicking the "Generate" button and open it with a convenient IDE. Add the Web3j dependency in the pom.xml: XML <dependency> <groupId>org.web3j</groupId> <artifactId>core</artifactId> <version>4.10.0</version> </dependency> 3. Creating a Smart Contract Let’s create the Counter.sol file as a Smart contract in your project: The first line in a contract sets the version of the source code that will be taken into account by the compiler: Plain Text pragma solidity ^0.8.20; For this example, we’ll create a contract with the Counter name using the contract keyword: Plain Text contract Counter { uint private count = 0; } We have declared the count state variable with data type uint as an unsigned integer. With the private access modifier, as in Java, this field can only be called within the current contract. Next, let’s add two functions for incrementing and decrementing the count variable. In addition, we add the getter method for the count variable with the view modifier that ensures the contract’s state won’t be changed: Plain Text pragma solidity ^0.8.20; contract Counter { uint private count = 0; function increment() public { count += 1; } function decrement() public { count--; } function getCount() public view returns (uint) { return count; } } 4. Compiling the Smart Contract and Creating a Web3j Wrapper To compile the listed Smart contract, we’ll use the previously installed solc library: Shell solcjs Counter.sol --bin --abi --optimize -o ../artifacts This will create two files with .abi and .bin extensions in the artifacts folder: To convert the generated .abi and .bin files into a Java class with method signatures from the contract, we’ll utilize the Web3j generator: Shell web3j generate solidity -b Counter_sol_Counter.bin -a Counter_sol_Counter.abi -o ../java -p com.example.smartcontract After execution, in the src/main/java/org/example directory, you should have a new class named Counter_sol_Counter. Let’s rename it to CounterContract: 5. Generating Ethereum Address To generate an Ethereum address and private key, you can use this website. Copy the private key and put it in the application.properties as ethereum.private-key property: I’ll be deploying our contract to the Sepolia testnet (you can also use Goerli or Kovan instead). In order to deploy a Smart contract and pay for transaction fees, we’ll need a little bit of ETH on the Sepolia testnet. Here are a few faucets where you can receive test coins: Sepolia Faucet Faucet Sepolia Sepolia PoW Faucet Infura Copy your Ethereum address and submit sending ETH coins: Lastly, we’ll add the URL of Sepolia's JSON-RPC endpoint in application.properties: Properties files ethereum.private-key=<your_ethereum_private_key> ethereum.provider=https://eth-sepolia.g.alchemy.com/v2/demo 6. Java Configuration Let’s create the Web3jConfig class, in which we declare beans for org.web3j.protocol.Web3j and org.web3j.crypto.Credentials using the parameters from application.properties: Java package com.example.smartcontract; import lombok.extern.slf4j.Slf4j; import org.springframework.beans.factory.annotation.Value; import org.springframework.context.annotation.Bean; import org.springframework.context.annotation.Configuration; import org.web3j.crypto.Credentials; import org.web3j.protocol.Web3j; import org.web3j.protocol.http.HttpService; @Configuration @Slf4j public class Web3jConfig { @Value("${ethereum.provider}") private String ethereumProvider; @Value("${ethereum.private-key}") private String ethereumPrivateKey; @Bean public Web3j web3j() { return Web3j.build(new HttpService(ethereumProvider)); } @Bean public Credentials credentials() { return Credentials.create(ethereumPrivateKey); } } Next, we declare a bean for the CounterContract, that will be deployed during initialization: Java @Bean public CounterContract counterContract() { CounterContract counterContract; try { counterContract = CounterContract.deploy(web3j(), credentials(), new DefaultGasProvider()).send(); // counter = Counter.load(counterContractAddress, web3j(), credentials(), new DefaultGasProvider()); } catch (Exception e) { log.error("Error while deploying a contract", e); throw new RuntimeException(e); } log.info("Counter contract has been deployed: {}", counterContract.getContractAddress()); return counterContract; } To use functions of the deployed Smart contract, let’s create the CounterContractService class with the injected CounterContract in it: Java package com.example.smartcontract; import lombok.SneakyThrows; import lombok.extern.slf4j.Slf4j; import org.springframework.beans.factory.annotation.Autowired; import org.springframework.stereotype.Service; import org.web3j.protocol.core.methods.response.TransactionReceipt; import java.math.BigInteger; @Service @Slf4j public class CounterContractService { @Autowired private CounterContract counterContract; @SneakyThrows public BigInteger getCount() { return counterContract.getCount().send(); } @SneakyThrows public void increment() { TransactionReceipt transactionReceipt = counterContract.increment().send(); log.info("increment transaction : {}", transactionReceipt.getTransactionHash()); } @SneakyThrows public void decrement() { TransactionReceipt transactionReceipt = counterContract.decrement().send(); log.info("decrement transaction : {}", transactionReceipt.getTransactionHash()); } } 7. Wrap Up At this point, the basic implementation of a Smart contract is ready. After launching the application, in the logs, you’ll see the address of the deployed contract, which you can trace in the Etherscan Explorer for Sepolia. The source code is available on GitHub.
AWS Lambda functions are a powerful tool for running serverless applications in the cloud. But as with any code, bugs can occur that can result in poor performance or even system crashes. Testing and debugging Lambda functions can help you identify potential issues before they become a problem. In this guide, we’ll cover everything from unit testing to automated tests to ensure that your Lambda functions are running smoothly. Testing and Debugging AWS Lambda Functions When it comes to AWS Lambda functions, testing and debugging are crucial steps in ensuring that the code runs smoothly. One of the essential concepts to understand is what a test event is in AWS Lambda. A test event is a simulated event that triggers the Lambda function. It allows you to test the function's response to different inputs. To create a test event, you need to define a JSON payload that resembles the actual event that will trigger the function. The payload should contain all the required parameters and data needed to execute the function successfully. You can then use this payload to test the function using the AWS Lambda console or any other testing framework. It's important to note that testing with different types of events is necessary to ensure that the Lambda function can handle various scenarios and inputs. By doing so, you can catch potential errors or bugs before pushing the code into production. In the next sections of this guide, we'll explore different methods for testing and debugging AWS Lambda functions to ensure they run as expected. Configure Test Event in AWS Lambda To configure a test event in AWS Lambda, you can follow a few simple steps. First, navigate to the AWS Management Console and select your Lambda function. In the Function code section, you should see a Test button. Clicking on this button will open up the Configure test event dialog box. From there, you can give your test event a name and then define the JSON payload that will be used to trigger the function. You can also use sample templates provided by AWS or upload your own JSON file. Once you've configured the test event, you can click the Create button to save it. Now, whenever you want to test your function, you can select the test event from the drop-down menu, and the function will be triggered using the specified payload. By configuring test events for different scenarios and inputs, you can ensure that your Lambda function is well-tested and performs as expected. Testing and Debugging AWS Lambda Locally via AWS SAM Another way to test and debug AWS Lambda functions is by using AWS SAM (Serverless Application Model) to run them locally. AWS SAM is an open-source framework that you can use to build serverless applications locally and deploy them to the cloud. With AWS SAM, you can write code in your favorite IDE (Integrated Development Environment), test it locally, and then deploy it to AWS Lambda. To get started with AWS SAM, you need to install it on your local machine and configure your development environment. Once you have done that, you can create a SAM template file that defines your Lambda function's configuration and dependencies. The SAM template file also specifies the events that will trigger your function. You can then use the sam local start-lambdacommand to test your function locally. This command simulates an event and passes it to your function for processing. You can also use sam local start-api to test your function as a REST API, which allows you to send HTTP requests to your function and receive responses. By testing and debugging AWS Lambda functions locally with AWS SAM, you can catch errors and bugs early on in the development process and streamline your deployment workflow. Setup Integration Tests With AWS Lambda Another important aspect of testing AWS Lambda functions is setting up integration tests. Integration tests are essential as they ensure that all the different components of your application work together seamlessly. AWS provides several tools and services that help you set up integration tests for your Lambda functions. One such tool is AWS Step Functions, which allows you to define a workflow that integrates multiple Lambda functions and other AWS services. You can use Step Functions to test the entire workflow end-to-end, including error handling and retries. Another tool is AWS CodePipeline, which provides a fully managed continuous delivery service that automates your release process for fast and reliable updates. In addition to these tools, you can also use third-party testing frameworks such as JUnit or TestNG to write integration tests for your Lambda functions. These frameworks allow you to create test suites that cover various scenarios and edge cases. When setting up integration tests, it is important to consider factors such as data consistency, error handling, and scalability. You should also ensure that your tests are repeatable and can be run in different environments. By setting up robust integration tests, you can catch any issues before they reach production and ensure that your Lambda function works as expected in real-world scenarios. How To Automate AWS Lambda Testing Automating AWS Lambda testing can save time and effort in the long run. One way to automate testing is by using AWS Lambda Layers. Lambda Layers is a distribution mechanism for libraries, custom runtimes, and other function dependencies. By creating a Lambda Layer for your testing framework and including it in your Lambda function, you can automate the testing process whenever there is an update to the function code. Another approach to automating AWS Lambda testing is by using AWS CodeBuild, a fully-managed continuous integration service that compiles source code, runs tests, and produces software packages. You can use AWS CodeBuild to automatically build and test your Lambda function whenever there is a new commit to the code repository. This ensures that any changes made to the codebase are tested thoroughly before being deployed to production. Finally, you can also use AWS CloudFormation to automate the deployment and testing of your Lambda function. AWS CloudFormation allows you to define infrastructure as code, including the Lambda function, its dependencies, and any associated resources. By defining a CloudFormation stack that includes your Lambda function and its tests, you can automate the entire deployment and testing process. In conclusion, automating AWS Lambda testing is crucial for ensuring the reliability and performance of your serverless applications. By using tools such as Lambda Layers, AWS CodeBuild, and AWS CloudFormation, you can streamline the testing process and catch any issues before they impact your users. Conclusion Testing and debugging are essential for ensuring that your AWS Lambda functions perform optimally. With the help of test events, you can create and manage different scenarios to check for bugs and errors before they become a problem. By using this proactive approach, you can rest assured that your Lambda functions will continue to run smoothly. Try out the steps outlined in this guide today to start testing and debugging your Lambda functions.
Much has been said about how much containerization has changed how we deliver and operate the software. But at the same time, the tools such as IDEs have changed just as much. We’ve gone from heavyweight, thick client tools running on our desktops with all the possible features we might need to be incorporated into the IDE to a very lightweight core. Everything is a plug-in, and increasingly the browser is the delivery platform for it. We have also seen a trend of IDEs becoming free, if not open source, and you may be surprised to read that Oracle has been at the forefront of this in the Java space. You have to look at the contributions to Netbeans as an Apache project. JDeveloper has been free, and Oracle has made plugins freely available for Eclipse for a very long time. This tradition continues with the code for the latest VS Code plugins being freely available (Oracle Samples — GitHub repository). While these products have provided plugin frameworks, as these products had a lot of the features for a language like Java built-in, the contributor community hasn’t been so broad. After all, the IDE out of the box contains a lot of tooling, and if you’re not working with the IDEs language, then you’ve got a significant footprint overhead that you won’t really utilize. Despite this, the more prominent software vendor-built plugins for the likes of Eclipse and Netbeans. For example, Oracle’s extensions to support work with PL/SQL, SOA Suite, and SonarSource have adapted their code quality tool for almost every IDE possible (more on this can be found here). They only provide the basics; plugging in everything else isn’t new. We can look at Emacs as an example of such an approach. But the modern approach has taken off with the likes of Sublime, Electron, and Eclipse Theia, which uses many of the same foundations as VS Code, such as the core Monaco editor (for more about the differences, check out this article). One of the most important details here is that Theia and VS Code can work with the same plugins (details here). This lighter nature and need for plugins to make the IDE more than just a fancy text editor. As there is no initial Language bias that came with earlier generations of IDE has meant the pool of potential users and plugin contributors is enormous. The one outlier to this trend has been JetBrains with IntelliJ IDEA and their other language variants. But here, the IDE has been packaged with the tooling for one language and variants of the base platform for different languages. Power of Plugins Supporting plugins and the core tool being made freely available has been core to the success of VS Code beyond the basic editor usability. As a result, a community of both independent developers and commercial vendors has developed a wealth of plugins. This is both a positive and a negative, as with all community-driven things, there are capability overlaps and varying quality as some solutions may not be more than beta solutions, and others lack sufficient support to sustain and maintain them. While download and star ratings can be indicative of quality and sustainability, this does make new arrivals into the marketplace harder to get established. Fortunately, if you’re working with a particular vendor, you can find their plugins grouped together in the Marketplace using the /publisher/<name>, for example, (and Oracle Labs). Some of the open-source communities, such as Apache, and MicroProfile Community, are identified as publishers. However, Linux Foundation and subsidiary groups like CNCF don’t appear to have their tools published collectively. Plugins From Cloud Vendors One newer trend with IDEs is to have plugins to make it increasingly easier not only to develop but also to ease the deployment into clouds such as Azure, GCP, and OCI. This means we can remain working within the IDE without having to drop out to command line tools or web interfaces, removing that context switching that can be annoying, particularly when working with limited screen estate or is' that have a greater emphasis on single app contexts. This can sometimes also include process simplification by combining several steps, such as generating the right packaging at the same time as pushing code. IDE Divergence While Microsoft has made VS Code freely available, it is still subject to Microsoft licensing; data gathering on its use goes to Microsoft as well as configuration and branding. VS Codium is, fortunately, the same as VS Code, but without those metrics, Microsoft license details, etc. There is a community that has taken to ensure the same source is packaged, deployable, and available through Brew, Chocolatey, Snap, and other package managers. You can, of course, build the binaries yourself. Not being bound to Microsoft’s implementation is great for those who are wary of what data may be collected or simply the solution being for a competitor, but it creates a new issue — specifically, the risk of divergence in the way things may be done. Most crucially, the plugins as this is the lifeblood of these types of IDEs. The Eclipse foundation has recognized this, and as is key to supporting Theia as an IDE intended to be a foundation that is extended on which people can build their own IDEs (examples of this include Red Hat OpenShift Dev Spaces). This is achieved by their Open VSX Registry, which provides a repository of plugins that are able to work across multiple IDEs, including VS Code and VS Codium, and any extension of Theia. Oracle, like all major developer tool providers, has gravitated to supporting VS Code but is also working to ensure that its development of plugins is compliant and visible on the Open VSX Registry. Releases will typically be published into the VS Code marketplace, as this enables the biggest single community. But verification and registration for VSX should follow quickly afterward. OCI Cloud Shell Editor It is also a reasonable expectation that these plugins will also appear in the OCI Cloud Shell Editor at some point in the future. Making the development processes even easier in cloud environments. We can this trend Conclusion When it comes to IDEs, VS Code and related solutions (such as Theia, VS Codium, etc.) are here, established, and with Github Codespaces and similar cloud vendor provider IDE solutions (such as Oracle’s Cloud Console Editor), they’re going to be around for a while for a good while. I think we’ll see the trend for cloud service plugins like those illustrated. What we’ve not yet observed is whether we’ll see services common to multiple cloud vendors, such as Kubernetes, OpenSearch, etc., coalesce around a common code base with differences down to naming conventions and authentication frameworks becoming pluggable. As to whether VS Code will become all dominant? Well, as we've discussed, VS Code has a strong open-source background, and there is standardization that aligns with this. It is going to be influential — the convenience and name recognition will combine with the pluggability characteristics will dominate. The adoption of common foundations into cloud Hosting solutions is also going to help keep it well positioned. But we do have some options in the form of VS Codium if you're concerned about undue influence.
Minikube: it's probably the simplest and the most approachable K8s cluster. As a lightweight K8s distribution designed for the purpose to run with low resources, an effective Minikube setup doesn't require anything else than your own laptop. From this perspective, Minikube is a great choice for development environments, able to give quick access to infrastructure elements like nodes, pods, deployments, services, and other K8s subtleties, more difficult to implement in a full-scale scenario. As a K8s-native runtime, Quarkus supports various types of clusters, including but not limited to Minikube, Kind, OpenShift, EKS (Elastic Kubernetes Service), AKS (Azure Kubernetes Service), etc. Packaging to Docker Images Quarkus offers several choices to package a cloud-native application based on different techniques and tools, as follows: Jib Docker S2I (Source to Image) In our prototype, we're using Jib, which, as opposed to the other two methods, has the advantage of not requiring a Docker daemon running on the host machine. In order to take advantage of it, just include the following Maven dependency in the master pom.xml file: XML ... <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-container-image-jib</artifactId> </dependency> ... Run a new build: Shell mvn -DskipTests -Dquarkus.container-image.build=true clean package When finished, if there is a Docker daemon running locally, the container image creation may be checked as follows: Shell $ docker images REPOSITORY TAG IMAGE ID CREATED SIZE aws-quarkus/aws-camelk-sqs 1.0.0-SNAPSHOT 63072102ba00 9 days ago 382MB aws-quarkus/aws-camelk-jaxrs 1.0.0-SNAPSHOT 776a0f99c5d6 9 days ago 387MB aws-quarkus/aws-camelk-s3 1.0.0-SNAPSHOT 003f0a987901 9 days ago 382MB aws-quarkus/aws-camelk-file 1.0.0-SNAPSHOT 1130a9c3dfcb 9 days ago 382MB ... Deploying to Minikube Our Apache Camel microservices don't require any modification or refactoring in order to be deployed to Minikube. However, the build process consisting of all the steps necessary for testing, packaging, and deploying the application to K8s is to be adapted to become cloud-aware and to take advantage of the Minikube peculiarities. Hence, the first modification that we need to tweak is to add the quarkus-minikube Maven artifact to our master pom.xml file, as shown below: XML ... <dependency> <groupId>io.quarkus</groupId> <artifactId>quarkus-minikube</artifactId> </dependency> ... This artifact will generate Minikube-specific manifest files in the project's target/kubernetes directory. As everyone knows, everything that K8s is about is described in YAML (Yet Another Markup Language) notation. And while K8s historically requires a quite heavy YAML authoring and editing process, using this artifact has the advantage of automatically generating the required YAML files or, at least, a base skeleton that might be enriched later. Performing a new build by running the mvn -DskipTests clean install command at the project's root level will produce two categories of files in the target/kubernetes directory for each Quarkus microservice: A kubernetes.yaml/json pair of files containing the manifest describing the microservice's K8 general resources A minikube.yaml/json pair of files containing the manifest describing the microservice's Minikube-specific resources For example, for the aws-camelk-jaxrs microservice, going to aws-camelk-jaxrs/target/kubernetes and opening the minikube.yaml file, you'll see that: XML ... spec: ports: - name: http nodePort: 30326 port: 80 protocol: TCP targetPort: 8080 selector: app.kubernetes.io/name: aws-camel-jaxrs app.kubernetes.io/part-of: aws-camelk app.kubernetes.io/version: 1.0.0-SNAPSHOT type: NodePort ... This manifest fragment defines a K8s service of the type NodePort listening for HTTP requests on the TCP port number 8080, mapped to the host's port number 30326. This configuration is Minikube specific as for other clusters like EKS, the type of the configured K8s service would be ClusterIP instead of NodePort. The selector paragraph defines the service name, version, and package, customized via the following properties: Properties files ... quarkus.kubernetes.part-of=aws-camelk quarkus.kubernetes.name=aws-camel-jaxrs quarkus.kubernetes.version=1.0.0-SNAPSHOT ... Another important point to notice is the AWS credentials definition. Our microservices need access to AWS and, for that purpose, some properties like the region name, the access key id, and value should be defined. While the region name isn't a piece of sensible information and may be defined as a clear text property, this isn't the case for the access key-related properties, which require to use K8s secrets. The following listing shows a fragment of the application.properties file: Properties files ... quarkus.kubernetes.env.vars.aws_region=eu-west-3 quarkus.kubernetes.env.secrets=aws-secret ... Here, the region name is defined as being eu-west-3 in plain text, while the AWS access key credentials are defined via a K8S secret named aws-secret. Running on Minikube We just have reviewed how to refactor our Maven-based build process in order to adapt it to K8s native. In order to run the microservices on Minikube, proceed as follows: Start Minikube Minikube should be installed, of course, on your box. That's a very easy operation; just follow the guide here. Once installed, you need to start Minikube: Shell $ minikube start ... $ eval $(minikube -p minikube docker-env) After starting Minikube, the last command in the listing above sets the K8s Docker registry to the instance running in Minikube such that the newly generated images be published directly to it. Clone the Project From GitHub Run the following commands to clone the repository: Shell $ git clone https://github.com/nicolasduminil/aws-camelk.git $ cd aws-camelk $ git checkout minikube Create a K8s Namespace and Secret Run the following commands to create the K8s namespace and secret: Shell $ kubectl create namespace quarkus-camel $ kubectl apply -f aws-secret.yaml --namespace quarkus-camel Here, after creating a K8s namespace named quarkus-camel, we create a K8s secret in this same namespace by applying the config in the manifest file named aws-secret.yaml, as shown below: YAML apiVersion: v1 kind: Secret metadata: name: aws-secret type: Opaque data: AWS_ACCESS_KEY_ID: ... AWS_SECRET_ACCESS_KEY: ... The properties labeled AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY are BASE64 encoded. Start the Microservices The same script (start-ms.sh) that we used in Part 2 in order to start the microservices may be used again for the same purposes. It has been modified, as shown below: Shell #!/bin/sh ./delete-all-buckets.sh ./create-queue.sh mvn -DskipTests -Dquarkus.kubernetes.deploy clean package sleep 3 ./copy-xml-file.sh Here, we start by cleaning up the environment and deleting all the S3 buckets named "mys3*", if any. Then we create an SQS queue named "myQueue" if it doesn't exist already. If it exists, we purge it by removing all the messages stored in it. The Maven command uses the property quarkus.kubernetes.deploy such that to deploy to Minikube the newly generated Docker images. Last but not least, copying an XML file into the input directory will trigger the Camel route named aws-camelk-file, which starts the pipeline. Observe the Log Files In order to follow the microservices execution, run the commands below: Shell $ kubectl get pods --namespace quarkus-camel $ kubectl logs <pod-id> --namespace quarkus-camel Stop the Microservices In order to stop the microservices, run the command below: Shell ./kill-ms.sh Clean up the AWS Infrastructure Don't forget to clean up your AWS infrastructure such that to avoid being billed. Shell $ ./delete-all-buckets.sh $ ./purge-sqs-queue.sh $ ./delete-sqs-queue.sh Stop Minikube Last but not least, stop Minikube: Shell $ eval $(minikube -p minikube docker-env --unset) $ minikube stop Enjoy! Previous Posts in This Series: Microservices With Apache Camel and Quarkus (Part 1) Microservices With Apache Camel and Quarkus (Part 2)
There are more than 250,000 companies/organizations around the world leaning on SharePoint to securely manage their most valuable documents, and more than 3 million total users. This widespread popularity makes the platform a market-leading document management solution - and this, by extension, makes it a worthwhile target for motivated threat actors. Bypassing SharePoint’s built-in security is an extremely difficult task, of course. The O365 environment provides tenants with powerful protection at every entry point, from exhaustive physical data center security up to leading-edge application security policies. Top-notch file encryption with SSL and TLS connections is applied to keep user data safe in transit, and BitLocker disk-level encryption with unique encryption keys is used to secure files at rest. Further, as infected file uploads have grown to become an extremely common attack vector, O365 provides built-in virus and malware detection policies (along with anti-phishing policies and various additional email link and attachment security measures) which can be customized extensively per individual or organizational tenants' needs. The list goes on, with each tenant's specific subscription level ultimately determining the extent of their built-in protection. As powerful as SharePoint's customizable built-in security policies are, however, no storage platform's policies are ever intended to be applied as a single point of protection for sensitive data. Document storage security, like any branch of cybersecurity, is a moving target requiring myriad solutions working together to jointly create a formidable defense against evolving attack vectors. In other words, any tenant’s threat profile can always be improved upon with selective layering of external security policies on top of built-in security policies. In the remainder of this article, I’ll demonstrate a free-to-use Virus Scanning API solution that can be integrated with a SharePoint Site Drive instance to scan files for viruses, malware, and a variety of non-malware content threats, working alongside O365's built-in asynchronous scanning to root out a wide range of file upload threat types. Demonstration The Advanced Virus Scan API below is intended to serve as a powerful layer of document storage security in conjunction with SharePoint's built-in customizable policies, directly scanning new file uploads in targeted Site Drive instances for a growing list of 17 million+ virus and malware signatures (including ransomware, spyware, trojans, etc.), while also performing full content verification to identify invalid file types and other non-malware threats hidden behind misleading file names and illegitimate file extensions. This API also allows developers to set custom restrictions against unwanted file types in the API request body, so various unnecessary and potentially threatening file types can be detected and deleted outright regardless of the legitimacy of their contents. For example, a Site Drive storing contract documents likely only requires common file types like .DOCX or .PDF: limiting files to these types helps minimize risks without compromising workflow efficiency. Below, I’ve outlined the information you’ll need to integrate this API with your SharePoint Online Site Drive instance, and I’ve provided ready-to-run Java code examples to help you structure your API call with ease. To start off, you’ll need to gather the following SharePoint information to satisfy mandatory parameters in the API request body: Client ID (Client ID access credentials; can be obtained from Azure Active Directory portal) Client Secret (Client Secret access credentials; also obtained from Azure Active Directory portal SharePoint Domain Name (i.e., yourdomain.sharepoint.com) Site ID (the specific SharePoint ID for the site drive you want to retrieve and scan files from) Optionally, you can also gather the following SharePoint information: Tenant ID (pertaining to your Azure Active Directory) File Path (path of a specific file within your Site Drive) Item ID (e.g., DriveItem ID) Once you’ve gotten all your mandatory information, you can start client SDK installation by adding the following reference to the repository in your Maven POM File (JitPack is used to dynamically compile the library): XML <repositories> <repository> <id>jitpack.io</id> <url>https://jitpack.io</url> </repository> </repositories> Then you can wrap up by adding the following reference to the dependency: XML <dependencies> <dependency> <groupId>com.github.Cloudmersive</groupId> <artifactId>Cloudmersive.APIClient.Java</artifactId> <version>v4.25</version> </dependency> </dependencies> At this point, you can add the imports and copy Java code examples to structure your API call: Java // Import classes: //import com.cloudmersive.client.invoker.ApiClient; //import com.cloudmersive.client.invoker.ApiException; //import com.cloudmersive.client.invoker.Configuration; //import com.cloudmersive.client.invoker.auth.*; //import com.cloudmersive.client.ScanCloudStorageApi; ApiClient defaultClient = Configuration.getDefaultApiClient(); // Configure API key authorization: Apikey ApiKeyAuth Apikey = (ApiKeyAuth) defaultClient.getAuthentication("Apikey"); Apikey.setApiKey("YOUR API KEY"); // Uncomment the following line to set a prefix for the API key, e.g. "Token" (defaults to null) //Apikey.setApiKeyPrefix("Token"); ScanCloudStorageApi apiInstance = new ScanCloudStorageApi(); String clientID = "clientID_example"; // String | Client ID access credentials; see description above for instructions on how to get the Client ID from the Azure Active Directory portal. String clientSecret = "clientSecret_example"; // String | Client Secret access credentials; see description above for instructions on how to get the Client Secret from the Azure Active Directory portal String sharepointDomainName = "sharepointDomainName_example"; // String | SharePoint Online domain name, such as mydomain.sharepoint.com String siteID = "siteID_example"; // String | Site ID (GUID) of the SharePoint site you wish to retrieve the file from String tenantID = "tenantID_example"; // String | Optional; Tenant ID of your Azure Active Directory String filePath = "filePath_example"; // String | Path to the file within the drive, such as 'hello.pdf' or '/folder/subfolder/world.pdf'. If the file path contains Unicode characters, you must base64 encode the file path and prepend it with 'base64:', such as: 'base64:6ZWV6ZWV6ZWV6ZWV6ZWV6ZWV'. String itemID = "itemID_example"; // String | SharePoint itemID, such as a DriveItem Id Boolean allowExecutables = true; // Boolean | Set to false to block executable files (program code) from being allowed in the input file. Default is false (recommended). Boolean allowInvalidFiles = true; // Boolean | Set to false to block invalid files, such as a PDF file that is not really a valid PDF file, or a Word Document that is not a valid Word Document. Default is false (recommended). Boolean allowScripts = true; // Boolean | Set to false to block script files, such as a PHP files, Python scripts, and other malicious content or security threats that can be embedded in the file. Set to true to allow these file types. Default is false (recommended). Boolean allowPasswordProtectedFiles = true; // Boolean | Set to false to block password protected and encrypted files, such as encrypted zip and rar files, and other files that seek to circumvent scanning through passwords. Set to true to allow these file types. Default is false (recommended). Boolean allowMacros = true; // Boolean | Set to false to block macros and other threats embedded in document files, such as Word, Excel and PowerPoint embedded Macros, and other files that contain embedded content threats. Set to true to allow these file types. Default is false (recommended). Boolean allowXmlExternalEntities = true; // Boolean | Set to false to block XML External Entities and other threats embedded in XML files, and other files that contain embedded content threats. Set to true to allow these file types. Default is false (recommended). String restrictFileTypes = "restrictFileTypes_example"; // String | Specify a restricted set of file formats to allow as clean as a comma-separated list of file formats, such as .pdf,.docx,.png would allow only PDF, PNG and Word document files. All files must pass content verification against this list of file formats, if they do not, then the result will be returned as CleanResult=false. Set restrictFileTypes parameter to null or empty string to disable; default is disabled. try { CloudStorageAdvancedVirusScanResult result = apiInstance.scanCloudStorageScanSharePointOnlineFileAdvanced(clientID, clientSecret, sharepointDomainName, siteID, tenantID, filePath, itemID, allowExecutables, allowInvalidFiles, allowScripts, allowPasswordProtectedFiles, allowMacros, allowXmlExternalEntities, restrictFileTypes); System.out.println(result); } catch (ApiException e) { System.err.println("Exception when calling ScanCloudStorageApi#scanCloudStorageScanSharePointOnlineFileAdvanced"); e.printStackTrace(); } To satisfy the request authentication parameter, you'll need to provide a free-tier API key, which will allow you to scan up to 800 files per month. Within this request body, you can set Booleans to apply custom non-malware threat policies against files containing executables, invalid files, scripts, password-protected files, macros, XML external entities, insecure deserialization, and HTML, and you can provide a comma-separated list of acceptable file types in the restrictFileTypes parameter to disallow unwanted file extensions. Any files violating these policies will automatically receive a CleanResult: False value in the API response body, which is the same value assigned to files containing viruses and malware. The idea is to enact 360-degree content protection in a single request so you can quickly delete (or quarantine/analyze) files that may pose a serious risk to your system. Below, I’ve provided a full example API response for your reference: JSON { "Successful": true, "CleanResult": true, "ContainsExecutable": true, "ContainsInvalidFile": true, "ContainsScript": true, "ContainsPasswordProtectedFile": true, "ContainsRestrictedFileFormat": true, "ContainsMacros": true, "VerifiedFileFormat": "string", "FoundViruses": [ { "FileName": "string", "VirusName": "string" } ], "ErrorDetailedDescription": "string", "FileSize": 0, "ContentInformation": { "ContainsJSON": true, "ContainsXML": true, "ContainsImage": true, "RelevantSubfileName": "string" } } It’s worth noting that regardless of how you choose to set your custom threat rules, files containing JSON, XML, or embedded images will be labeled as such in the API response as well.
The article will cover the following topics: Why is Envoy proxy required? Introducing Envoy proxy Envoy proxy architecture with Istio Envoy proxy features Use cases of Envoy proxy Benefits of Envoy proxy Demo video - Deploying Envoy in K8s and configuring as a load balancer Why Is Envoy Proxy Required? Challenges are plenty for organizations moving their applications from monolithic to microservices architecture. Managing and monitoring the sheer number of distributed services across Kubernetes and the public cloud often exhausts app developers, cloud teams, and SREs. Below are some of the major network-level operational hassles of microservices, which shows why Envoy proxy is required. Lack of Secure Network Connection Kubernetes is not inherently secure because services are allowed to talk to each other freely. It poses a great threat to the infrastructure since an attacker who gains access to a pod can move laterally across the network and compromise other services. This can be a huge problem for security teams, as it is harder to ensure the safety and integrity of sensitive data. Also, the traditional perimeter-based firewall approach and intrusion detection systems will not help in such cases. Complying With Security Policies Is a Huge Challenge There is no developer on earth who would enjoy writing security logic to ensure authentication and authorization, instead of brainstorming business problems. However, organizations who want to adhere to policies such as HIPAA or GDPR, ask their developers to write security logic such as mTLS encryption in their applications. Such cases in enterprises will lead to two consequences: frustrated developers, and security policies being implemented locally and in silos. Lack of Visibility Due to Complex Network Topology Typically, microservices are distributed across multiple Kubernetes clusters and cloud providers. Communication between these services within and across cluster boundaries will contribute to a complex network topology in no time. As a result, it becomes hard for Ops teams and SREs to have visibility over the network, which impedes their ability to identify and resolve network issues in a timely manner. This will lead to frequent application downtime and compromised SLA. Complicated Service Discovery Services are often created and destroyed in a dynamic microservices environment. Static configurations provided by old-generation proxies are ineffective in keeping track of services in such an environment. This makes it difficult for application engineers to configure communication logic between services because they have to manually update the configuration file whenever a new service is deployed or deleted. It leads to application developers spending more of their time configuring the networking logic rather than coding the business logic. Inefficient Load Balancing and Traffic Routing It is crucial for platform architects and cloud engineers to ensure effective traffic routing and load balancing between services. However, it is a time-consuming and error-prone process for them to manually configure routing rules and load balancing policies for each service, especially when they have a fleet of them. Also, traditional load balancers with simple algorithms would result in inefficient resource utilization and suboptimal load balancing in the case of microservices. All these lead to increased latency, and service unavailability due to improper traffic routing. With the rise in the adoption of microservices architecture, there was a need for a fast, intelligent proxy that can handle the complex service-to-service connection across the cloud. Introducing Envoy Proxy Envoy is an open-source edge and service proxy, originally developed by Lyft to facilitate their migration from a monolith to cloud-native microservices architecture. It also serves as a communication bus for microservices (refer to Figure 1 below) across the cloud, enabling them to communicate with each other in a rapid, secure, and efficient manner. Envoy proxy abstracts network and security from the application layer to an infrastructure layer. This helps application developers simplify developing cloud-native applications by saving hours spent on configuring network and security logic. Envoy proxy provides advanced load balancing and traffic routing capabilities that are critical to run large, complex distributed applications. Also, the modular architecture of Envoy helps cloud and platform engineers to customize and extend its capabilities. Figure 1: Envoy proxy intercepting traffic between services Envoy Proxy Architecture With Istio Envoy proxies are deployed as sidecar containers alongside application containers. The sidecar proxy then intercepts and takes care of the service-to-service connection (refer to Figure 2 below) and provides a variety of features. This network of proxies is called a data plane, and it is configured and monitored from a control plane provided by Istio. These two components together form the Istio service mesh architecture, which provides a powerful and flexible infrastructure layer for managing and securing microservices. Figure 2: Istio sidecar architecture with Envoy proxy data plane Envoy Proxy Features Envoy proxy offers the following features at a high level. (Visit Envoy docs for more information on the features listed below.) Out-of-process architecture: Envoy proxy runs independently as a separate process apart from the application process. It can be deployed as a sidecar proxy and also as a gateway without requiring any changes to the application. Envoy is also compatible with any application language like Java or C++, which provides greater flexibility for application developers. L3/L4 and L7 filter architecture: Envoy supports filters and allows customizing traffic at the network layer (L3/L4) and at the application layer (L7). This allows for more control over the network traffic and offers granular traffic management capabilities such as TLS client certificate authentication, buffering, rate limiting, and routing/forwarding. HTTP/2 and HTTP/3 support: Envoy supports HTTP/1.1, HTTP/2, and HTTP/3 (currently in alpha) protocols. This enables seamless communication between clients and target servers using different versions of HTTP. HTTP L7 routing: Envoy's HTTP L7 routing subsystem can route and redirect requests based on various criteria, such as path, authority, and content type. This feature is useful for building front/edge proxies and service-to-service meshes. gRPC support: Envoy supports gRPC, a Google RPC framework that uses HTTP/2 or above as its underlying transport. Envoy can act as a routing and load-balancing substrate for gRPC requests and responses. Service discovery and dynamic configuration: Envoy supports service discovery and dynamic configuration through a layered set of APIs that provide dynamic updates about backend hosts, clusters, routing, listening sockets, and cryptographic material. This allows for centralized management and simpler deployment, with options for DNS resolution or static config files. Health checking: For building an Envoy mesh, service discovery is treated as an eventually consistent process. Envoy has a health-checking subsystem that can perform active and passive health checks to determine healthy load-balancing targets. Advanced load balancing: Envoy's self-contained proxy architecture allows it to implement advanced load-balancing techniques, such as automatic retries, circuit breaking, request shadowing, and outlier detection, in one place, accessible to any application. Front/edge proxy support: Using the same software at the edge provides benefits such as observability, management, and identical service discovery and load-balancing algorithms. Envoy's feature set makes it well-suited as an edge proxy for most modern web application use cases, including TLS termination, support for multiple HTTP versions, and HTTP L7 routing. Best-in-class observability: Envoy provides robust statistics support for all subsystems and supports distributed tracing via third-party providers, making it easier for SREs and Ops teams to monitor and debug problems occurring at both the network and application levels. Given its powerful set of features, Envoy proxy has become a popular choice for organizations to manage and secure multicloud and multicluster apps. In practice, it has two main use cases. Use Cases of Envoy Proxy Envoy proxy can be used as both a sidecar service proxy and a gateway. Envoy Sidecar Proxy As we have seen in the Isito architecture, Envoy proxy constitutes the data plane and manages the traffic flow between services deployed in the mesh. The sidecar proxy provides features such as service discovery, load balancing, traffic routing, etc., and offers visibility and security to the network of microservices. Envoy Gateway as API Envoy proxy can be deployed as an API Gateway and as an ingress (please refer to the Envoy Gateway project). Envoy Gateway is deployed at the edge of the cluster to manage external traffic flowing into the cluster and between multicloud applications (north-south traffic). Envoy Gateway helped application developers who were toiling to configure Envoy proxy (Istio-native) as API and ingress controller, instead of purchasing a third-party solution like NGINX. With its implementation, they have a central location to configure and manage ingress and egress traffic and apply security policies such as authentication and access control. Below is a diagram of Envoy Gateway architecture and its components. Envoy Gateway architecture (Source) Benefits of Envoy Proxy Envoy’s ability to abstract network and security layers offers several benefits for IT teams such as developers, SREs, cloud engineers, and platform teams. Following are a few of them. Effective Network Abstraction The out-of-process architecture of Envoy helps it to abstract the network layer from the application to its own infrastructure layer. This allows for faster deployment for application developers, while also providing a central plane to manage communication between services. Fine-Grained Traffic Management With its support for the network (L3/L4) and application (L7) layers, Envoy provides flexible and granular traffic routing, such as traffic splitting, retry policies, and load balancing. Ensure Zero Trust Security at L4/L7 Layers Envoy proxy helps to implement authentication among services inside a cluster with stronger identity verification mechanisms like mTLS and JWT. You can achieve authorization at the L7 layer with Envoy proxy easily and ensure zero trust. (You can implement AuthN/Z policies with Istio service mesh — the control plane for Envoy.) Control East-West and North-South Traffic for Multicloud Apps Since enterprises deploy their applications into multiple clouds, it is important to understand and control the traffic or communication in and out of the data centers. Since Envoy proxy can be used as a sidecar and also an API gateway, it can help manage east-west traffic and also north-south traffic, respectively. Monitor Traffic and Ensure Optimum Platform Performance Envoy aims to make the network understandable by emitting statistics, which are divided into three categories: downstream statistics for incoming requests, upstream statistics for outgoing requests, and server statistics for describing the Envoy server instance. Envoy also provides logs and metrics that provide insights into traffic flow between services, which is also helpful for SREs and Ops teams to quickly detect and resolve any performance issues. Video: Get Started With Envoy Proxy Deploying Envoy in k8s and Configuring as Load Balancer The below video discusses different deployment types and their use cases, and it shows a demo of Envoy deployment into Kubernetes and how to set it as a load balancer (edge proxy).
I'm continuing my journey of getting more familiar with HTTP APIs by reading related RFCs. This time, I read the Health Check Response Format for HTTP APIs at the suggestion of Stefano Fago. In this post, I'd like to summarize my reading. Note that it's a draft. Moreover, it has been dormant for nearly two years and, thus, has been automatically expired. However, it's the closest to a specification on health checks and thus deserves some love. Sample Data Visualization Even though it's not a long read, it's a bit "dry." Fortunately, the specification offers a JSON sample. I copy-pasted it in PlantUML, and presto, it shows a visual representation of it: Let's have a look at the proposed structure element by element. The Root Object At its simplest, the response is a JSON object with a mandatory status property: Values can be: pass for healthy status: The value can also be ok (for NodeJS) and up (for Spring Boot) to account for existing health check libraries. The HTTP status code must be in the range from 2xx to 3xx. warn for healthy status but with concerns with the same HTTP status range fail to indicate unhealthy status: possible alternative values include error (NodeJS) and down (Spring Boot). The HTTP status code must be in the range from 4xx to 5xx. One can add additional optional values: version: Public version of the service releaseId: Internal version of the service; for example, the version would be incremented for non-compatible change, while the releaseId could be the commit hash or a semantic version serviceId: The unique identifier of the service description: self-explanatory notes: Array of non-structured notes output: Plain error message in case of pass or warn.; the field should be left blank for pass The links Objects The links object consists of object pairs. Values are URIs, while keys can be URIs or common/registered ones: see RFC5988 for common values; e.g., self. The checks Objects Keys of checks objects consist of two terms separated by a colon, component name, and measurement name. The latter can be either: A pre-defined value: utilization, responseTime, connections, or uptime A standard term from a well-known source; e.g., IANA, microformat.org, etc. A URI Values consist of one of the following keys: componentId: unique id of this component componentType: A pre-defined value, component, datastore, or system A standard term from a well-known source; e.g., IANA, microformat.org, etc. A URI observedValue: Any valid JSON value observedUnit: Unit of measurement status: As the parent object's status, but for this component, only affectedEndpoints: If the component is not pass, lists all affected endpoints time: Date-time in ISO8601 format of the observation output: As the parent object's output, but for this component, only links: See the previous section Any other non-standard value I tried implementing the above with Spring Boot using a custom HealthIndicator. Here's the best I could come up with: The current structure of the JSON response needs to be (easily?) customizable. You'd need to create your endpoint. I hope the Spring Boot team provides the option to generate a compatible structure. Conclusion The Healthcheck IETF draft is a great initiative to standardize health checks across the industry. It would allow monitoring tools to rely on HTTP status and response body without ad-hoc configuration on each service. Unfortunately, the draft is expired because of a lack of activity. I'd love to see it revived, though. To Go Further: "Spring Boot Actuator: Health"