Remote Troubleshooting on the Shop Floor

Steady manufacturing productivity drives industrial companies to look for new strategies that assure maximum machine availability. Though this is not an easy task, there are efforts to reduce downtime by implementing new strategies to optimise the troubleshooting procedure. One popular strategy relies on remote troubleshooting, which allows manufacturer experts and maintenance technicians (in-house or from other plants) to access the machines.

A Fictional Situation

The machine operator calls the maintenance technician for support. The clock is ticking. The technician uses his long experience to address the machine fault. After two hours and going through the maintenance best practice manual, the maintenance department decides that this is an unknown fault in the machine, so it’s time to call the manufacturer. After a lengthy call, the manufacturer sends a technician overseas. Express flight time = 16–20 hours to reach the destination. The technician finds the fault on-site and reports back to headquarters to address the new fault in new machines. The machine is operating again after two days.

This example shows: The consequences of this incident are a long downtime, products not delivered to the customer, stressful situation for the maintenance and production departments, and even worse, customer (dis)satisfaction, and (unexpected) additional costs for special logistics in meeting the deadlines. Remote troubleshooting helps to address this situation.

Let’s look at the topic in detail.

Remote Troubleshooting

Remote troubleshooting reduces time by allowing the manufacturer experts and the maintenance technicians (e.g. in-house or from other plants) to access the machines. So, the expert performs a machine diagnostic, identifies the problem, and recommends a solution. As a result, the reaction time is reduced, and consequently, the downtime.

But technically, let’s connect the field laptop to the programmable logic controller (PLC) of the machine and start a remote desktop application for the manufacturer expert or the maintenance technician. Voilà, problem solved. Well, not quite, some aspects need to be considered:

• IT compliance with the solution, especially connection security and user management. Established virtual point-to-point connections provide a secured channel through virtual private network (VPN) protocols (e.g. Secure Shell VPN, internet protocol security (IPsec), SSL/TLS). Furthermore, the secured channel or tunnel is authenticated (e.g. passwords, biometrics, two-factor authentication (2FA)).
• Network segmentation. Suppose there is only one network for the office devices and the shop floor machines. A remote desktop application could be an open door for an attack vector (e.g. malware, phishing, distributed-denial-of-service (DDoS)) or for (unwanted) access to other corporate IT resources.
• Communication interface to the machine means the machine controller should be accessible (e.g. some machines only offer connectivity via proprietary communication protocols).
• User level agreement between the plant and the internal/external users, which includes a protocol on the use of the service and coordination with production management to stop the machine(s).
• User access management, which establishes user permissions for users and limits access to the desired machine(s).

An Example Architecture using VPN

So far, we have reviewed the advantages of remote troubleshooting and the main points that should be considered. It looks like a lot of technical aspects, a bit complicated indeed. But, as the saying goes, “a picture is worth a thousand words”, what about a diagram showing all the parties involved? The following diagram illustrates a typical setup for remote troubleshooting:

The elements in orange represent the remote troubleshooting components used for the VPN:

• The remote connection device on-site is connected to the same machine network. The remote connection may have the capability of network segmentation, which allows separating the machine networks of manufacturer “X” and “Y”, with the definition of “VLAN 1” and “VLAN 2”, respectively.
• The remote connection hosting server establishes the tunnel between the remote connection device and the VPN client. This server can be self-hosted by the company or belong to a service provider.
• The VPN client is running on the laptop/PC of the manufacturer expert or the maintenance technician (in-house or from another plant).

The elements in red represent the machine components of manufacturer X. They can be an industrial PC (IPC), a human machine interface (HMI), switches, PLCs, or other field devices (e.g. measurement devices, data loggers, intelligent sensors, camera systems).

Considerations before the Implementation

The company should review these technical factors before selecting a remote troubleshooting solution:

• The automation controllers should have a layer-3 connectivity (OSI Model) (e.g. TCP/IP). In the case of older machines, the company can perform a retrofit on the controllers, which means a middle device that can translate the machine communication protocol (e.g. Profibus, MODBUS) to TCP/IP.
• A network segmentation should separate the office network from the shop floor network in the first instance. Therefore, it is highly recommended to implement separate networks per machine following IEC 62443 standard. This network segmentation between machines allows granting the manufacturers access to their machines. Furthermore, it prevents undesired access to other company resources on the network (e.g. OPC-UA servers, databases, manufacturing execution system (MES)).
• An information technology (IT) / operational technology (OT) review of the proposed architecture should be implemented so the new solution conforms with the IT policies.
• An OT technical check of the machines that should be accessible. A list of critical production equipment can support this operation.
• Responsibility definition for the administration and maintenance of the remote troubleshooting solution (e.g. software updates, hardware check, changes in the architecture, hardware / software ordering)

Selection of a VPN-based Remote Troubleshooting Solution

There are technical factors that the company needs to consider for a VPN selection:

• Who will host the remote connection server (e.g. either a self-hosted server by the company or a third company)? Where will the server be hosted? The last question is linked to the data protection policy of the company.
• Must the remote connection device have a network segmentation capability?
• Which IP security protocol needs to be implemented?
• Must the remote connection device have a network address translation (NAT) capability?
• Must the device be for industrial purposes, or will it be kept in a special cabinet?

Next Steps

Secured remote troubleshooting can reduce downtime on the shop floor. However, it also has other benefits: the company can interconnect services located in different places (overseas) and even have centralised access to the data. Though this extra benefit was not the primary motivation for remote troubleshooting, it has a massive impact on further adding value to the business. This interconnection can ease the data integration of the different systems (e.g. machines, process, quality, energy) for further data analysis and predictive models to assess the machine operator. For this reason, the company must analyse the engineering requirements for the remote connection and the machine data collection. In this manner, the company can design the technical infrastructure and make an investment that can cover both aspects, thus avoiding future (unnecessary) ad-hoc upgrades for machine data collection.

Our experts for IoT/IIoT, smart products, digital production and data sciences are happy to support you in your projects. Would you like to learn more about IoT and our services in this area?

Contact us directly

You can find more exciting topics from the adesso world in our blog posts published so far.