First Principle of Software Engineering

Aristotle’s account of a first principle (in one sense) as “the first basis from which a thing is known” (Met. 1013a14–15) – Wikipedia

Thinking about the first principle of software engineering as the essence of the practice helps us understand how to align our expectations and choose strategies that are appropriate. It also helps us think of Software Engineering as a discrete discipline that requires skill and experience. It is not a shallow practice producing lines of code in a vacuum. It is more like producing well thought out lines of code to complete a valuable idea for solving a problem. This value depends on various attributes to be fit to be a part of or a whole solution. Further, value in software is often not a single point in time. Value multiplies over time. Short-sighted additions die on the vine and require twice the work to revive later.

Therefore, the essence of Software Engineering is to:

Resiliently Add Computing Value

It is easy to see that disposable code does play a role in the IT landscape. However, it has less value to a Software Engineer. The first principle indicates a goal of accumulating value in a solution or system through modification, preferably enhancement and not repair. Although, if you are inheriting a system that has more value in the idea than execution, you are forced to start at a disadvantage.

Baseline First Principle Strategies

After stripping it down to its essence, we can identify the most fundamental Software Engineering strategies. These are Resilience, Completeness, Transparency and Automation.

Let us expand on these concepts and explain how they may relate to security concerns.

Resilience

Resilience stands out as an obvious first choice given it is found in the first strategy statement. Resilience describes an application’s ability to keep running predictably under unfavorable circumstances and load. It also describes how software should fail gracefully without corrupting or exposing data or causing any other damage. As well as it indicates recovery from such failure.

You may recognize that Resilience as a strategy for Availability. It is also how we deal with injection, timing attacks, enumeration and various other attacks that leverage glitching. We will cover various tactics to address Resilience, like Defensive Coding and Dependency Management.

Completeness

Another baseline strategy supporting the first principle is Completeness. This describes addressing implicit requirements in a way that creates a cohesive solution. Each context brings with it certain attributes that must be addressed. Sometimes this involves complementary solutions as in the case of Authorization and Authentication to support multi-user contexts. Further, authorization would completely cover sensitive data without “insecure direct object references”.

Completeness means we maintain confidentiality and integrity. We will cover aspects of Completeness with the sections on Auth, Square, Requirements Mastering and Testing.

Transparency

Transparency is something that has become particularly important in development processes, but it is invaluable at the code level. There are certain situations that require being specific and explicit. Rarely, if ever, is there value in obscurity, especially when speaking of security. Further, a transparent system is a supportable system.

Transparency describes the need to enforce non-repudiation. It can be used to guide what we need to log. Transparency helps us measure how resilient the system is.

Automation

In a sense, automation is the most basic of computing principles. It is why computing is valuable. It is also important to modern process management. Automation explains the need to document and version control processes. For many, automation enables the “always be practicing” philosophy that is a precursor to Continuous Integration and Continuous Deployment. Further automation limits a process’s exposure to failures due to differences between executions.

Ultimately, Automation is a first-level strategy shared directly with security. The ability to repeat a process with highly controlled change between executions is nearly priceless if the process is transparent enough that failures can be accounted for in subsequent execution. This allows an iterative approach to complete problem domain coverage. Resilience can also be enhanced as new and unforeseen circumstances emerge. This focuses effort on progress without wasting it on activities that would otherwise need repeating.

Conclusion

I encourage you to use the First Principle to develop a organizational mindset that embraces these stratigies at their most fundamental level.

In a time when we are on the cusp of a code explosion in much the same way we had an explosion of networks with the popularity of cloud, it is important to establish this complete Software Engineering mindset. Once such an understanding is positioned, we can build strategies for quality secure solutions to distinguish ourselves from the deluge of low brow software that is knocking at our door.

Integrating a Security Culture

A common security issue I see is that Security Teams simply have trouble getting traction. That is, they do all kinds of analysis and testing, but development teams never directly address the changes that need to be made.

Some may be aware of their need for security but may not understand the role of a Security Team. A Security Department does not typically configure servers, network equipment, or contribute product code. Also, Security Team members may not comprehend that Security does not typically make business decisions.

As some have experienced, saying that security is everyone’s job can only have a limited effect, even with full cooperation of the whole business. A much more effective situation is if Security has been strategically integrated with the culture.

Strategy

Deciding the relationship between Software Engineering and Application Security clearly and according to the desired culture will make it easier to be embraced by the culture. This should be done as early as possible and communicated up-front during employee onboarding right beside the message that ‘Security is everyone’s job’.

Application Security should strive to supply information that is readily available for Product and Architecture to make decisions. They should strive to supply guidelines and tooling for Governance of the Development process. They should also ensure the production security posture is known to the stakeholders and Development itself.

In many organizations, Security members do not contribute code to the product. On the surface, being a non-conde-contributing team can keep the compliance lines clear for things like Separation of Concerns. I have seen organizations where any member is allowed to contribute code. This can build a strong developer centric culture, but it also requires a strong process with proper pre-merge reviews and tooling in place to ensure compliance with policies, standards, and guidelines.

Some organizations will recognize that Security Professionals will need to be highly trained in Security and that Application Security is a specialization requiring a significant investment of ongoing learning. Also, Application Security for a software developing organization is a further specialization of that. It is also closely related to Quality as an Assurance practice. AppSec Engineers will usually not be Software Engineering experts, nor Software Architects because of the broad experience and training already required of them. Therefore, they should not be making specific Software Engineering or Software Architectural decisions, although they bring strong supporting expertise and data.

Effective

Being clear about who is responsible for designing and implementing can have several effects that you should be aware of. Firstly, some Security people have fallen prey to the fallacy that the only way to effectively make change is for Security to ride roughshod over programmers. Explicitly placing authority for change in the more qualified hands of Sofware Engineers will frustrate some individuals in Security.

If you have experienced the situation where Security rides roughshod over programmers you have seen how it makes Security a disruption because it shortcuts any established processes and best practices including sidestepping Product, Architecture and often any sort of Engineering Design. As many of us know, disruption is the primary enemy of every coder. In such a situation it makes Security a willful enemy of progress.

On the positive side, explaining this early can allow those with inexperience to observe how the process works to get things done. They will also see how they are relieved of the stress of responsibility when they do not agree with the action or lack of action taken as the communication and risk acceptance can be traced through the process to be carried on many shoulders. This then builds on the fellowship of the team cooperation over time and gives both Security and Development the opportunity to appreciate the strength of each other’s craft.

Once this is established, when a Security Incident happens, “breaking glass” is much more effective because of clear roles and trust among the teams. The long-term effect is then also established as the result of the emergency can have a known path back into the process and has less of a chance of regressing in the future.

Takeaway

Having a security culture beyond ‘Security is Everyone’s Job’ requires the recognition of the strengths each part of the organization brings. A process established on this recognition will feel natural, despite the very deliberate efforts to get it there.

Starting an Application Security Program

Sometimes the hardest step to take is the first. If you are facing a non-existent Applicaiton Security (AppSec) Program, or one that needs a reboot, here are some suggestions on how/where to start.

Beginners Advantage

The first mistake that can be made when there is no formal AppSec Program is to fail to take advantage of the existing situation. If you have no program, the Development Team are the only ones responsible for security. Long-standing programs struggle to establish this because they have long since taken Security out of the hands of the Development Team.

The reality of an AppSec Program is that its primary function will always be Assurance. Rarely, if ever, have I observed an AppSec Engineer directly contribute to the application lifecycle in any other way. Many do not even have Software Engineering experience as they are usually plucked from Infrastructure Security. This is corroborated by the many job listings for AppSec, which list Penetration Testing skills and certifications as requirements.

Because this background is common, development teams are rarely recognized for established security practices. Any existing Development Team will be performing activities devoted to fixing flaws and reacting to incidence as well as many other activities essential to the success of an AppSec Program.

Assertion

By taking advantage of this initial strength in Assurance, an AppSec program can assert itself as an aid to achieve higher standards. The focus of those higher standards is obviously Security, but also with an understanding that broader attributes of quality and maintainability are directly related to repeatable security enhancement. There is an obligatory relationship between Security and Quality, where Application Security Programs can never productively be more sophisticated than Quality Assurance (QA) and Control (QC).

Based on this approach, the AppSec Team should identify and acknowledge areas where the Development Team are doing well. Then they should work together to identify ways in which small changes can be made to emphasize security or to bring existing practices into full compliance with security definitions.

Footprinting

Security frameworks can be highly valuable in the initial stages of an AppSec program. These come with tools for framing context, setting goals and tracking progress. Frameworks are not a one-size-fits-all solution. Development teams will differ wildly. Learning from implementing a security framework will forever influence them, good and bad. So be careful about forcing a framework into a situation where it does not fit.

One personal favorite is OWASP SAMM. It establishes a vocabulary and definition system that can be socialized among development teams. It also has a questionnaire that can be used to footprint the existing development process and then be used to track progress. Early on SAMM can be used to identify practices that partly match the purpose defined and signify low investment gains toward maturing the Security Program.

Establishing Roots

Certain domains of OWASP SAMM require heightened interaction between AppSec and Dev teams. Those are the domains of Design and Verification. Verification is a SAMM domain that can be initiated and driven though automated tooling until the output reaches a level where more manual Verification processes can be implemented to find less-common issues.

To quickly establish a valuable feedback flow from Verification, implement automated Application Security Testing tools. Then create a process for adding Development work items from these tools. This should closely model how flaws from QA flow. It may be necessary to enhance the QA process or even the development process to accommodate multiple sources.

It is good to note that up to this point progress reporting to upper management should consist entirely of the progress of Application Security implementation. It should have nothing to do with vulnerabilities or flaws until the Development Team has had several cycles to get acquainted with the information and how to address these sorts of issues. There should also be an additional timebox to work through some initial issues. This additional buffer should be worked into the plan early. This places the AppSec Team in an adversarial posture toward vulnerabilities and not the Development Team.

Breaking Ground

Once the Verification process has started it is time to establish elements from the SAMM Design domain to direct the implementation away from common pitfalls and major catastrophe. This may be where establishing the AppSec Program gets difficult, especially if there is a lack of development background on the Security Team. This is a much simpler task when the previous steps have been performed in concert with the Development Team. It may be that when done well, the Development Team will seek out the AppSec Team with the desire to integrate security methodologies in the Requirements, Architecture and Design phases of development.

At this point the team can simply start by asking questions to introduce security to requirements gathering or backlog grooming as well as architecture. It is also useful to introduce Threat Modeling via the Four Question Framework:

• What are we working on?
• What can go wrong?
• What are we going to do about it?
• Did we do a good job?

Growth

This is the point where meaningful conversations can happen about the AppSec Program’s maturity. Next, work on the SAMM domains of Operations and Governance to bolster a holistic Application Security Program. Start scoring using the OWASP SAMM questionnaire on a regular interval to track the program’s progress.

I hope this inspires new Application Security Programs to progress down a path of logical development with an eye to embedded cooperation with Development. There are times when working in Security can seem hopeless unless we look ahead to maturing the methodology. It is not about “Shifting Left” to force reinvented processes on development. Never forget that Software Development was established long before someone decided to excavate security concerns and assign them to AppSec. Reintegrating them is the best way to gain quick traction and lasting productivity.

Oracle Named Parameters for SQL Injection in .Net

Injection has dropped in priority for many in recent years. For example, on the OWASP Top 10 it went from #1 in 2017 to #3 in 2021. However, Injection continues to be an issue throughout the software development world. If you look at what replaced it as the #1 concern on the OWASP Top 10 you would be excused if you thought things have gotten worse instead of better.

Personally, I see input handling as having several distinct levels. There is the interface level, where the input received from the source should be validated and filtered. At the other end of the process there is the destination or the ‘sink’. Like the source, which has many contexts to influence how it is handled, the destination could have many contexts that should influence how data is filtered and escaped before being passed to the sink. This means that there are always at least two contexts to be concerned about.

Being mindful about how we take input and use it is important. I have encouraged using Input Interaction Modeling in the past because it helps avoid Injection pitfalls. This should lead a Software Engineer to contemplate the components and methods involved. There are some components that should simply never be custom built. Cryptography is an example of a component that should be used and not built one of a kind. Another example is filter and escaping routines for your database. Why? Because there is a mechanism that does it for you already.

Oracle Bind Parameters

For Oracle there is the bind parameter. A parameter is just like it sounds. You put a placeholder in your query for a value to be bound to. At the most simplistic level, this allows you to execute a query multiple times without redefining the query. From a performance perspective, not only does the .Net runtime like this at scale because you avoid operations like string concatenation, but Oracle (and other RDMS) sees a performance boost as well (TL;DR- bind variable = prepared statement = cache boost).

So, what does this have to do with filtering and escaping? When binding to the variable the value is escaped according to the specific RDMS being used, in this case Oracle. Taking this further, you can set the DbType on the bind variable which will throw an exception if the value cannot be converted. This all happens before the query is sent to the database. Thus, you get filtering and escaping for free simply by using a core feature of ADO.NET.

The Code

So what does this look like?

// C#
using (var connection = new OracleConnection(ConfigurationManager.ConnectionStrings["OracleExpress"].ConnectionString))
{
    var query = "SELECT SOME_COLUMN, ANOTHER_COLUMN, THIRD_COLUMN FROM SOME_TABLE WHERE ANOTHER_COLUMN = :SomeParam AND THIRD_COLUMN = :AnotherParam";
    var command = new OracleCommand(query, connection) { CommandType = CommandType.Text, BindByName = true };
    command.Parameters.Add(":AnotherParam", OracleDbType.Varchar2).Value = "Ping";
    command.Parameters.Add(":SomeParam", OracleDbType.Varchar2).Value = "Foo";
    connection. Open();
    var reader = command.ExecuteReader();
    while (reader.Read())
    {
        // Do things
    }
}

' Visual Basic
Using connection As OracleConnection = New OracleConnection(ConfigurationManager.ConnectionStrings("OracleExpress").ConnectionString)
    Dim query As String = "SELECT SOME_COLUMN, ANOTHER_COLUMN, THIRD_COLUMN FROM SOME_TABLE WHERE ANOTHER_COLUMN = :SomeParam AND THIRD_COLUMN = :AnotherParam"
    Dim command As OracleCommand = New OracleCommand(query, connection) With {.CommandType = CommandType.Text, .BindByName = True}
    command.Parameters.Add(":AnotherParam", OracleDbType.Varchar2).Value = "Ping"
    command.Parameters.Add(":SomeParam", OracleDbType.Varchar2).Value = "Foo"
    connection.Open()
    Dim reader As IDataReader = command.ExecuteReader()
    While reader. Read()
        ' Do things
    End While
End Using

So, let’s break this down. Notice :SomeParam in the query text? That is a named parameter. Then once the command is instantiated, a value is bound to that parameter name via Parameter.Add().

I would like to say ‘it is that simple’ but Bind Parameters do not solve all problems. Some systems need to define dynamic table names, and some add or remove columns on the fly. Those situations must be handled carefully. However, even in those circomstances there is every reason to use Bind Variables in ALL SQL.

How much of your code is not using Bind Variables? I challenge you: eliminate all string concatenation in SQL queries in all code.

The Level 2 Game

Yes, Software Finish is a team effort. If one contributor is outputting Level 5 code and another is playing “The Level 2 Game”, everyone pays the price. The Level 2 Game is that of a “sandbagger”.

It goes something like this: A contributor implements features ignoring implicit requirements and with the full expectation that there are flaws that will be finished and fixed later. When asked to make minor adjustments this type of contributor starts negotiating against the change because what they do is hard to change. Also, when they do not understand something, they will proceed regardless since they have the expectation of exploiting the lack of code ownership so that someone else will make sure it works.

So how do you deal with a “Level 2 Gamer”. My preferred method is to put someone capable of Level 4 or 5 Finish on their team and enforce their sign off on all Pull Requests. This will force action on the part of the sandbagger. They will either understand, learn, and get better, or get frustrated and move on unfortunately. How do you deal with these sorts of contributors?

Software Finish Levels

Craftsmanship in software is important. It is important beyond simple quality and productivity. Lets consider some information about the neglected craftsmanship of Three Mile Island.

Three Mile Island

The control room: “investigators found key indicator lights on the backs of panels, switches out of calibration, and tags covering warning panels, and never fewer than 52 alarms blinking at all times. Over 100 alarms would sound when the reactor started to fail in Pennsylvania, and it was impossible to sort the critical ones from the usual.” – Three Mile Island – America’s Worst Nuclear Accident

As a casual observer it seems to me that they were skipping design steps, neglecting maintenance, and outright skipping tasks to do other things. Sound familiar?

There are many standards for developing software, quality, and testing. Many are difficult to grasp, and some are full of ceremony. What I would like to do is offer something that is light, and therefore admittedly a little subjective.

The Software Finish Level Matrix

Software Finish Levels

What Software Finish Level do you target? What do you output? Eventually I would like to craft more qualitative definitions and a simple questionnaire to help us be honest with ourselves.

Just what is it? It is made up of six levels that describe the form of the code. This has nothing to do with user interface or experience or anything to do with graphic design. This has nothing to do with the maturity of the code. It has everything to do with existing and new code. It describes code readability and organization on a function and process level. It describes how decisions are made about the code.

From these statements about the Software Finish Level, we can start to formulate a basis for being deliberate about the “fitness” of the software. If you want to sponsor a triathlon winner, don’t go looking for a lazy person with no desire to run. You start with someone fit from the beginning. Fitness being the software’s ability to do what is intended.

Like not waiting till race day to see how well your sponsored candidate can run the triathlon, you will want to be sure of their performance beforehand if you want a winner. With software this is done with the various Software Assurance practices. The primary concerns for assurance are performance quality and security. Software Finish Levels describe how deliberately these results were achieved and how much effort will be needed to address areas that failed to satisfy assurance expectations. The less deliberate the team was the less likely the team is to repeat satisfactory results or address failures.

Software Finish is a team activity. While it is easier to achieve Level 5 Finish with contributors acting at that level, we know that it takes time for someone to achieve this level of skill and craftsmanship. To achieve the best outcomes with various skill levels, teams must support each other and be willing to learn and follow some leadership structure. More on that to come…

What does good Secure Software Engineering training look like?

Because I have been so vocal about the lame state of training for developers concerning security, several have asked if I would put together some suggestions for good AppSec Training.

The fundamental attribute of effective Secure Development Training is that it focuses on applying Software Engineering best practices to solve security concerns. Knowing how a tester searches for issues may be entertaining for some, but that is not how to solve the issue. An early warning of this is if there is a mention of making programmers “think like hackers”.

Secure Sofware Engineering requires several secure foundational elements. Logging is a great subject to start with because it serves as a context for conversations around data and security leading to PII and PHI and other concerns.

So here is my list, let me know if you can think of other things, I am sure I did not get them all.

• Logging
• Code Review
• Input Interaction Modeling
• Defensive Coding
• Implicit Security Requirements
• What is the importance of consistency and standards?
• Change Surface Management
• Why are reusable modules important in security solutions?
• Dependency Management
• Investigating Open Source to assess project viability
• When to build it yourself or not (ex. Leftpad vs Encryption)
• Trust Boundaries (ex. how to treat the browser, HTTP, API…)
• Authentication/Authorization schemes (RBAC, OAuth, Claims, JWT…)
• Secure Ci/CD – protect your product from malicious change
• Software Engineering’s role in Incident Response

In exceptional circumstances, some Software Engineers and Architects may be interested in things like Secure System Design/Architecture, Threat modeling and Encryption. Things that could indicate the training neglects Engineering and Craftsmanship solutions to Security is mention of:

• MITRE ATT&CK Framework
• Penetration Testing or “Hacking”
• TOP 10 lists

Neglecting Software Engineering and Craftsmanship leads to heavy recurrence of the same class of vulnerability and the same flaws. This is because they are often hyper localized code corrections. Often, security tickets find a way of shortcutting proper process and scrutiny. The only way to change or prevent these common mistakes is to have good training in the first place, and the right leadership in place to keep the momentum going.

Input Interaction Modeling

As a Software Engineer, it is appropriate to consider each input for more than threats. Many Software Engineers do this already without extra effort. However, it may only be informed by their isolated experiences. This post will examine some formalization to enrich this existing practice. This way, with a little practice we can significantly improve our initial software security posture and overall quality.

Input

We will call the process “Input Modeling”. The goal is to understand the value, its context, and usage. This can take place to a limited degree at design time. However, this is often the responsibility of the programmer at the time of implementation. Please note this is explicitly not a security practice, even though there are security implications, as with any engineering activity.

There are many similarities between human and application interfaces when it comes to input. At the most fundamental level most input is either data to be reflected to the user (human or application) or values that are used to carry out tasks like flow control.

Text

Text may seem to be complex to take as input and act on. However, when this must be done, it is good to examine how the input is to be used. If the data is to be computed, the scope of what is allowed is automatically reduced. It is easy to get caught in the snare of creating strict rules that make enhancement and maintenance difficult. However, normalizing the text (ex. ToUpper()) and comparing values to expected input before propagating the value as a strong type or Enum value without direct conversion is a solid method of extracting reliable actionable values.

When the data is something like a formatted blogpost for the user, it is important to encode and escape it appropriately for a particular persistence. Improper escaping can result in data loss or corruption. Then as the data leaves the code instance to go elsewhere, it must likewise be escaped or encoded for that particular production. That is to say that the encoding must be decided upon at the point in code that the output destination is known. No method of processing should be considered to be universal, or data could lose integrity and cause issues on the receiving end.

It is helpful to be consistent for each context and avoid multiple implementations for each. More paths lead to less adoption. It is important to make it clear what is expected when processing the data. This leads to a quicker turnaround when flaws are later found.

Numbers

Numbers can be deceptively simple to deal with. When the value is typed by a user or the transport of the input text the conversion back to a number can introduce issues when not done carefully. It is good to avoid creating custom code for these conversions as the framework likely has something already.

The second thing that should be considered is the bounds of the number used. Even if the number is big, or the size of the datatype, set it to a max if it is over your value. The stability gained is worth having to recompile when this sort of bound needs changing.

Whenever possible the smallest type should be used for the variable taking this input. Setting realistic bounds on the value of that number can also prevent later performance issues. It should also be considered if any numbers less than zero are needed. If not reject them or use absolute values.

The bottom line is to take what will be used and leave everything else to pass from the memory of the request. Thoroughly examine the system’s use of the data and be specific about what is accepted.

Complex Types

Fundamentally speaking, all values are built on these. For this reason and in the context of the system itself, further examination should be done on how this input value will interact and be combined with other values. If methods of serialization are to be used, safety should be given at least the same importance as performance and ease of implementation. Further constraints may need to be employed prior to these interactions to ensure the integrity of the outcome.

Successfully accepting input in a consistent manner is the same as an animal learning to walk. If they are not willing and capable of doing this, they are a liability to the herd and will be culled by predators. In the software world, this liability doesn’t go away when a coder does, they persist in code until attacked. This attack could be a clumsy user or an attacker after your valuable data. Either of these could result in instability of the application and loss of credibility and money.

Not Threat Modeling

If you are familiar with threat modeling, you may be able to see the relationship with Input Modeling. In the context of input, if adequate Input Modeling happened many threats would be ineffective without the need to consider threats specifically. Further Threat Modeling is often done without intimate working knowledge of the system. Because of this, the exercise would need to start with widely known existing threats forming less than creative threat theories. Then extra work is required to investigate the threat theories.

…But First

Before you leave the Input Model, once you determine how to consistently receive it cleanly from the user, determine the sensitivity of that value. This is a small thought exercise with huge value. Ask: Does it identify a person? If so, how directly does it identify a person (think PII, PHI and GDPR)? How would the user feel if someone else saw it? Is there regulation around how it is used? Should you audit changes made to it?

Data sensitivity is something that should be considered from the beginning because it should determine what is done with the value. This level of detail does not always emerge from the architectural step.

When I get a chance, I will publish a checklist to help you build your security sensitivity.

Web API Security Considerations

This is the second post in a series on Web APIs and Security. Why should Web APIs matter to security? What makes them different?

First, they have been around for a long time, and they are not going to go away. So, it is not something that can be ignored and there is plenty of legacy folly to go around.

Secondly, since a Web API is easy to expose and consume through many flexible means, it is also easy to do so without understanding everything exposed. Nearly anything can interact with a Web API, so the caller should not be trusted to send clean data or even act the right way, not even after an Auth process, not even on a trusted network.

Then, there is the fact that they are meant to be automation friendly. This means that any cobbled together bot can quickly examine the attack surface of an API, probe for more information and attack with little investment from the bot’s human owner.

In addition, because any function exposed can be directly called, all functions must enforce auth and properly handle raw user input; two things that are often glossed over in Software Engineering education and training. In complex Web API offerings, it is easy to see how areas can be neglected or missed entirely. For some, just having an inventory of what is there seems difficult.

A Dash of Unlearn

When software engineers enter the workplace, their more experienced coworkers are usually buried with their day-to-day grind that does not explicitly include herding new talent. While this is a fundamental flaw in many business’s software engineering cultures, it leads to teams creating rules to address common mistakes programmers make early on. This is itself a mistake. One made by instructors all the time as well.

Some examples are: “If you have more than 7 parameters in your constructor, you need to divide it into two classes.” or “No fields, auto-properties only.” or my favorite “Never use sequential Ids.” The issue with all these rules (especially that last one) is that they often hide the real issue without giving an opportunity to explain why. What is worse is when these arbitrary rules came from somewhere outside engineering. From good intending security workers for example.

Then there are the rules that clearly came out of a catastrophic failure of some sort, that someone handed down the command to “Never, ever, ever do X”. These find their way into software engineering and stink up the culture. Don’t do it. An ironic rule for how to treat rules is: if there is not a clear time to break the rule, it is probably invalid. If you come across a blanket rule, the effort in finding out why and when to break the rule will be well worth your time.

Dismantle – an exercise in mature engineering

There are better ways to work through the intricacies of protecting APIs, but first lets dismantle one of those rules we were talking about: “Never use sequential identifiers.”

Personally, I think this one continues to proliferate because it is something even the most non-technical person can superficially understand. It has even made it into high profile documents and findings descriptions.

The basis for this rule seems to be simplicity itself, if not elegance: People and machines can count. Therefore, if I can count to the next number I can guess the id of the next object, even if it is not mine. Assuming the system does not want you to access certain objects.

However, the fact that you have the ID, regardless of how you got it, should not give you access. The system must manage this authorization. It must be acknowledged that an ID can be obtained via shoulder-surfing, various browser and man-in-the-middle attacks, in addition to guessing. Therefore, the act of guessing is not the whole exploit, which also means it is not the mitigation.

The mitigation is to properly (read securely) verify the authorization of the requestor to access the resource in the requested way. This is a suitable time to point out that there is a significant amount of implicit functionality to do so. These Implicit Requirements are specific to the platforms and technologies involved in the Web API.

“All good?”

Does this mean it is always ok to use sequential IDs? No. Without a broad rule, how do you determine if it is ok to use a numeric and sequential ID? It helps to understand why sequential IDs are used in the first place. One may assume that it is simply convenient, but that would be oversimplifying.

Outside of a relational database, sequential IDs may be difficult to reliably implement. Because of the high performance of numeric indexes, relational data management systems (RDMS) usually offer a way to quickly create unique number generating fields. The most performant way to do this is to generate numbers sequentially. So, it is convenient to simply use this contextually unique identifier. More importantly, it offers performance much greater than full text indexes or even special GUID indexes that some RDMS support.

Ideas are more powerful than guns. We would not let our enemies have guns, why should we let them have ideas?

From the perspective that anyone can understand, seeing a number signals the possibility of sequence. This can motivate anyone curious to test and see if that is true. If an attacker puts in another number in the sequence and is granted access to values they are not authorized to view the Web API is said to have Broken Object Level Authorization. As the name indicates, the sequential id is not a vulnerability. The vulnerability is that the requester is not authorized to access the data (aka the Object).

For the sake of discussion, imagine the insecurity is fixed. In this case, what would be gained by not using sequential IDs? If the system would deny them access (with a ‘not found’ error) the attack would fail. On one hand we wasted an attacker’s time. On the other hand, we got them drooling. This could be seen as a net wash. You may be concerned that you got an attacker’s attention. It is unfortunate that poor software engineering has tutored attackers that this is a straightforward way to get at what should otherwise be inaccessible. Be assured it is possible to teach them the opposite.

When considering performance, the situation could be that the performance hit between sequential id and GUID/UUID is worth avoiding attracting attention. Or it may be possible to use a sequential ID for referential integrity within the relational data and another unique identifier in the Web API for a hybrid approach. This is how some object databases store data.

Not All IDs Have the Same Value

What if, for example, a sequential primary key might be used as the ‘Customer Number’ which may be printed on statements and used over the phone. This makes them both more sensitive and easier to obtain. It is good practice to have a well-defined, single purpose for each value, especially IDs.

Using a primary key as a ‘Customer Number’ could cause issues when performing data migration or schema changes. Object IDs (often called Primary Keys in relational databases) have the purpose of referential integrity and are used by the application. Other identifiers with the purpose of being used outside the system should be generated separately and be secured as an object value. URIs are logged as a reference in many situations in Web APIs. If the purposes are not kept separate it could lead to sensitive identifiers being leaked via standard logging.

To sum it all up, there is a case for not using sequential identifiers. However sequential identifiers are not themselves a vulnerability. Therefore, an absolute ban on them driven by security brings the solution to a problem that does not exist. Software Engineering should drive the decision, weighing performance with exposure with cost of engineering effort.

TL;DR

In many cases Software Engineering has been plagued by rules that hide genuine issues and rob coders of the opportunity to learn.

Where significant code is being created the Software Industry is slow to mature when it comes to Security. This is despite significant advances in quality and process. The Application Security space is broken and steadily alienates itself from the rest of the process. Using buzzwords like DevSecOps is not going to fix it. It is time for development teams to bring security home to where it belongs.

Next

What about Threat Modeling? Unfortunately, Threat Modeling is inadequate as a software engineering practice. As a Software Engineer, it is more appropriate to model values in the broader sense of usage, using standards handling those situations. Many Software Engineers do this already without thinking too much about it. However, it may only be informed by one’s own experiences. Next, we will add some formalization to broaden and inform this exercise. This way, with a little practice we can significantly improve our initial software security posture.

Web APIs And Security

The security industry marketers recently found a three-letter-acronym useful in pushing their wares. With this follows a panic among those without understanding. The TLA is API. If you are reading this, you probably want an accurate picture before we bite the bare hook that vendors are dangling. So, let’s define API, look at the threats, and define some best practices for building them to be dependable and secure. In later posts there will be some musings about where vendors can help.

If you are not a developer, or even if you know what an API is, for this and other discussions, it will be framed first in this series.

What is API?

The acronym API stands for Application Programming Interface. You may be familiar with a User Interface or UI. A UI is a way for a user/person to interact with software. As the name suggests, an API is an interface that allows a developer to program an application against. That is to say: it is an interface for other software to use.

What is an interface? It is a point of interaction. Like a electrical outlet is a standard shaped place for electronics to interact with electricity. A doorknob is a standard shaped way for humans to interact with a door controlling access to a room.

Often in sales as well as in the English language generalized terms can sometimes be abused in such a way that the meaning becomes fluid. This is often exploited by sales to shoehorn their solution into your org verbally and then leave you to fill in the gaps. To that end, I want to emphasize that the term API refers to a concept that has no restriction to platform, transport protocols, data types, the Web, or the Internet. The fact is that APIs predate the Internet. The concept of an API is two pieces of code interacting with each other and exchanging data.

API refers to a concept that has no restriction to platform, transport protocols, data types, the Web, or the Internet. An API is two pieces of code interacting with each other and exchanging data.

So where does the security risk enter the picture? What could one do with anything that would raise its security profile? Give it the maximum exposure possible. A way to do this would be to build the API so it was available on the Internet. Then use a widely known protocol and standard to document everything about the API that can be interacted with. There are several specific things this could describe, but I specifically mean a Web API with an easily accessible OpenAPI Document.

API Exposure

A Web API is as exposed as it gets, and for good reason. The advantage of a Web API is that it can communicate over the Internet using standard Web technologies. Other methods of API consumption may have entailed referencing DLLs and possibly distributing all the functionality in binary form with the consuming application. Yet other remote API methods may have involved implementing your own version of a proprietary protocol.

Referencing the API over the Web allows for greater flexibility in consumer platforms and convenient consumer onboarding. Several incompatible platforms can share functionality if they talk the HTTP Standard (RFC 2616). Not only do they share code, but code instances and the workloads can be managed, maintained, and scaled in a uniform way. In short: less duplication, less waste.

Security

How does this exposure matter to security? One could say the data is already passing between the browser and server on websites now. Web APIs are no different than webpages when it comes to data. However, there are details about Web APIs that merit special consideration. Let’s look further at Security Considerations.